Photoelectric conversion device, photoelectric conversion system, mobile body, and apparatus

Abstract

This photoelectric conversion device comprises: a plurality of photoelectric conversion units that each output photon detection signals in response to the incidence of photons; a plurality of counters that each count photon detection signals from a corresponding photoelectric conversion unit; and a computation unit that performs computation processing that is based on the count value from each of the plurality of counters. The computation unit is provided with: a holding unit that holds count values from the plurality of counters; and a vector calculation unit that, on the basis of count values from the counters at a first point in time and count values held by the holding unit that are count values from the counters at a second point in time earlier than the first point in time, calculates a vector value representing an optical flow between the second point in time and the first point in time.

Description

Photoelectric conversion device, photoelectric conversion system, mobile object and equipment

[0001] The present invention relates to a photoelectric conversion device, a photoelectric conversion system, a mobile object, and an apparatus.

[0002] In recent years, automobiles, drones, and the like have been equipped with numerous image sensors, one of the purposes of which is to acquire optical flow (also called "motion vectors"). Optical flow is used, for example, in simultaneous self-localization and environmental map generation (SLAM: Simultaneous Localization and Mapping). In general, to acquire optical flow, arithmetic processing using the Lucas-Kanade method, block matching, or the like is performed on video data (a sequence of images captured at a predetermined interval). Patent Document 1 discloses an optical flow sensor configured to collect optical flow algorithm parameters.

[0003] Special Publication No. 2021-520699

[0004] There is a demand for acquiring optical flow faster and with less latency than ever before. However, in the configuration described in Patent Document 1, although algorithm parameters required for calculating optical flow are extracted within the sensor, data processing outside the sensor is also required, and it is not possible to acquire optical flow within the sensor at high speed.

[0005] An object of the present invention is to provide a photoelectric conversion device that can acquire optical flow at high speed without requiring data processing by an external device.

[0006] According to one disclosure of the present specification, there is provided a photoelectric conversion device having a plurality of photoelectric conversion units, each outputting a photon detection signal in response to the incidence of a photon, a plurality of counters, each counting the photon detection signal of a corresponding photoelectric conversion unit, and a calculation unit that performs calculation processing based on the count values ​​of each of the plurality of counters, wherein the calculation unit has a holding unit that holds the count values ​​of the plurality of counters, and a vector calculation unit that calculates a vector value representing the optical flow between the second time and the first time based on the count value of the counter at a first time and the count value of the counter at a second time earlier than the first time, which count value is held by the holding unit.

[0007] Furthermore, according to another disclosure of this specification, there is provided a photoelectric conversion device comprising: a plurality of photoelectric conversion units, each outputting a photon detection signal in response to the incidence of a photon; a plurality of counters, each counting the photon detection signal output from a corresponding one of the photoelectric conversion units; and an arithmetic unit that performs arithmetic processing based on the count values ​​of the plurality of counters, wherein the arithmetic unit periodically acquires count values ​​from the plurality of counters while the counters are performing a series of counting operations, outputs vector values ​​representing optical flow based on the acquired count values ​​at a first period, and outputs the count values ​​of the counters at a second period longer than the first period.

[0008] Furthermore, according to yet another disclosure of the present specification, there is provided a photoelectric conversion device comprising: a plurality of pixels, each having a photoelectric conversion unit that outputs a photon detection signal in response to incidence of a photon, and a counter that counts the photon detection signal output from the photoelectric conversion unit; and a calculation unit that calculates a vector value representing an optical flow between a first time and a second time, based on a count value of the counter for the plurality of pixels at the first time and a count value of the counter at a second time different from the first time, wherein the plurality of pixels are divided into a plurality of pixel blocks, each including two or more of the pixels, and arranged to form a plurality of rows and a plurality of columns, a calculation unit is provided for each of the pixel blocks, adjacent calculation units arranged in the same pixel block row or pixel block column are connected to each other, and the vector value output from one of the calculation units is transferred via another of the calculation units arranged in the same pixel block row or pixel block column.

[0009] According to the present invention, it is possible to realize a photoelectric conversion device that can acquire optical flow at high speed without requiring data processing by an external device.

[0010] FIG. 1 is a block diagram showing a schematic configuration of a photoelectric conversion device according to a first embodiment. FIG. 2 is a perspective view showing a configuration example of a pixel of the photoelectric conversion device according to the first embodiment. FIG. 3 is a block diagram showing a configuration example of a calculation unit of the photoelectric conversion device according to the first embodiment. FIG. 4 is a diagram showing a configuration example of a sub-pixel block in the photoelectric conversion device according to the first embodiment. FIG. 5 is a timing chart showing a method for driving the photoelectric conversion device according to the first embodiment. FIG. 6 is a block diagram showing a configuration example of a calculation unit of a photoelectric conversion device according to a second embodiment. FIG. 7 is a block diagram showing a configuration example of a calculation unit of a photoelectric conversion device according to a third embodiment. FIG. 8 is a block diagram showing a schematic configuration of an imaging system according to a fourth embodiment. FIG. 9 is a diagram showing a configuration example of an imaging system according to a fifth embodiment. FIG. 10 is a diagram showing a configuration example of a moving body according to the fifth embodiment. FIG. 11 is a block diagram showing a schematic configuration of equipment according to a sixth embodiment.

[0011] [First Embodiment] A photoelectric conversion device and a driving method thereof according to a first embodiment of the present invention will be described with reference to Figs. 1 to 6. Fig. 1 is a block diagram showing a schematic configuration of a photoelectric conversion device according to this embodiment. Fig. 2 is a perspective view showing an example configuration of a photoelectric conversion device according to this embodiment. Fig. 3 is a circuit diagram showing an example configuration of a pixel of a photoelectric conversion device according to this embodiment. Fig. 4 is a block diagram showing an example configuration of a calculation unit of a photoelectric conversion device according to this embodiment. Fig. 5 is a diagram showing an example configuration of a sub-pixel block in a photoelectric conversion device according to this embodiment. Fig. 6 is a timing chart showing a method for driving a photoelectric conversion device according to this embodiment.

[0012] 1, the photoelectric conversion device 100 according to this embodiment includes a pixel / calculation unit 10, a signal processing unit 40, an output interface unit (output IF unit) 50, and a control unit 60. The pixel / calculation unit 10 includes a plurality of pixels P arranged in a plurality of rows and a plurality of columns, and a plurality of calculation units C provided for each pixel block including a predetermined number of pixels P.

[0013] The pixel / calculation unit 10 may be provided with a plurality of pixels P arranged in a two-dimensional array of, for example, M columns and N rows. In Fig. 1, the reference symbol P of each pixel is accompanied by coordinates (m, n) representing the column number m and row number n. Here, M and N are integers equal to or greater than 1, m is an integer equal to or greater than 0 and less than M, and n is an integer equal to or greater than 0 and less than N.

[0014] The calculation unit C is a functional block that performs calculation processing based on output signals from pixels P. One calculation unit C may be provided for each pixel block containing a predetermined number of pixels P. Here, each pixel block may contain two or more pixels P arranged in I columns by J columns. One of I and J may be an integer greater than or equal to 2, and the other may be an integer greater than or equal to 1. For example, in the example shown in FIG. 1, one calculation unit C is provided for each pixel block of 4 columns by 4 rows. In FIG. 1, coordinates (m', n') are added to the reference symbol C of the calculation unit to distinguish them. When a calculation unit C is provided for each pixel block of I columns by J rows, m' is a multiple of I, and n' is a multiple of J. For example, the calculation unit C corresponding to a 4 columns by 4 rows pixel block containing pixels P(0,0) and P(3,3) at the diagonal corners is calculation unit C(0,0). The calculation unit C corresponding to the 4 column by 4 row pixel block including pixels P(4,4) and P(7,7) at the diagonal corners is calculation unit C(4,4). A predetermined number (e.g., 16) of pixels P included in one pixel block are connected to the calculation unit C provided corresponding to that one pixel block. Note that in this specification, a group of rows corresponding to a pixel block may be referred to as a pixel block row, and a group of columns corresponding to a pixel block may be referred to as a pixel block column.

[0015] A control line 12 is arranged in each row of the pixel / calculation unit 10, extending in a first direction (the horizontal direction in FIG. 1 ). Each control line 12 is connected to each of the pixels P arranged in the corresponding row and serves as a signal line common to these pixels P. Furthermore, a plurality of control lines 14 are arranged in the pixel / calculation unit 10, extending in the first direction. Each control line 14 is connected to each of the calculation units C that share the same row coordinate (n′), and serves as a signal line common to these calculation units C. For example, one control line 14 is connected to calculation units C(0,0) and C(4,0), and another control line 14 is connected to calculation units C(0,4) and C(4,4). The first direction in which the control lines 12 and 14 extend may be referred to as the row direction or the horizontal direction. Each of the control lines 12 and 14 may include a plurality of signal lines for supplying a plurality of types of control signals to the pixels P or the calculation unit C. The control lines 12 and 14 of each row are connected to the control unit 60.

[0016] The calculation unit C(m',n') is connected to the calculation unit C(m',n'-J) and calculation unit C(m',n'+J) adjacent to it in a second direction (the vertical direction in FIG. 1) that intersects with the first direction. Furthermore, the calculation unit C(m',N-J) arranged at the end in the second direction is connected to the signal processing unit 40. This allows the calculation results of each of the calculation units C(m',0) to C(m',N-J) to be transferred in the column direction to the signal processing unit 40. The second direction in which data from the calculation unit C is transferred may be referred to as the column direction or the vertical direction.

[0017] The signal processing unit 40 performs predetermined signal processing on the data received from the calculation unit C and outputs the data to the output IF unit 50. The output IF unit 50 has, for example, a serializer that converts parallel data into serial data and an external interface circuit, and is a circuit for transmitting the data after signal processing by the signal processing unit 40 to a system external to the photoelectric conversion device 100.

[0018] The control unit 60 is a control circuit that generates control signals for controlling the operations and timings of the pixel / arithmetic unit 10, the signal processing unit 40, and the output IF unit 50, and supplies them to each functional block.

[0019] The photoelectric conversion device 100 of this embodiment may be configured such that all of the above-described functional blocks are arranged on a single substrate, or may be configured as a stacked photoelectric conversion device in which the above-described functional blocks are fabricated separately on multiple substrates and then bonded together to form an electrical connection. The stacked photoelectric conversion device may be configured by bonding two substrates together, or by bonding three or more substrates together. The electrical connection between the multiple substrates can be achieved using a technique called TSV (Through Silicon Via), in which a metal member is embedded in a hole that penetrates one of the multiple substrates and an insulating layer, and then this metal member is connected to the wiring layer of the other substrate. Another form of electrical connection between the multiple substrates can be hybrid bonding, in which multiple insulating layers and multiple metal portions are bonded together at the bonding surfaces where the insulating layers of the multiple substrates are bonded together. The bonded metal portions are typically made of copper. These bonded metal portions are connected to the wiring layers of each substrate. This allows for electrical connection between the multiple substrates. These electrical connection methods are merely examples, and various other methods for electrically connecting a plurality of substrates, such as bonding using microbumps, can be employed.

[0020] FIG. 2 is a schematic diagram of a photoelectric conversion device 100 constructed by stacking a pixel substrate 110 and a circuit substrate 120. For example, pixels P, which are components of the pixel / calculation unit 10, can be arranged on the pixel substrate 110. For example, calculation unit C, signal processing unit 40, output IF unit 50, and control unit 60, which are components of the pixel / calculation unit 10, can be arranged on the circuit substrate 120. In this case, the pixels P and calculation unit C can be arranged in overlapping regions in a planar view, as shown in FIG. 2, for example. By arranging the pixels P and the other components on separate substrates, it is possible to reduce the size and improve the functionality of the photoelectric conversion device 100 without sacrificing the light-receiving area of ​​the pixels P.

[0021] Next, an example configuration of a pixel P will be described with reference to Fig. 3. Each pixel P may be configured with a photoelectric conversion element PD, a switch unit MP, a waveform shaping unit 16, and a counter 18, as shown in Fig. 3, for example. The photoelectric conversion element PD may be configured with an avalanche photodiode (hereinafter referred to as "APD"). The switch unit MP may be configured with a P-type MOS transistor. The photoelectric conversion element PD, switch unit MP, and waveform shaping unit 16 form a photoelectric conversion unit that outputs a photon detection signal in response to incident photons.

[0022] The anode of the APD constituting the photoelectric conversion element PD is connected to a node to which a voltage VL is supplied. The cathode of the APD constituting the photoelectric conversion element PD is connected to the drain of the P-type MOS transistor constituting the switch unit MP. The source of the P-type MOS transistor constituting the switch unit MP is connected to a node to which a voltage VH is supplied. The input node of the waveform shaping unit 16 is connected to a connection node (node ​​A) between the photoelectric conversion element PD and the switch unit MP. The output node of the waveform shaping unit 16 is connected to an input node of the counter 18. The output node of the counter 18 is connected to the calculation unit C corresponding to the pixel P. A signal CLKB, which is an inverted signal of the clock signal CLK, is input from the control unit 60 via the control line 12 to the gate of the P-type MOS transistor constituting the switch unit MP. A reset signal RES for resetting the count value is input from the control unit 60 via the control line 12 to the control node of the counter 18.

[0023] The photoelectric conversion element PD may be configured as an APD, as described above. The voltages VL and VH are set so that a reverse bias voltage sufficient for the APD to perform avalanche multiplication is applied. In one example, a negative high voltage is applied as the voltage VL, and a positive voltage similar to the power supply voltage is applied as the voltage VH. For example, the voltage VL is approximately −30 V, and the voltage VH is approximately 1 V. By supplying a reverse bias voltage sufficient for avalanche multiplication to the APD, the charge generated by light incident on the APD undergoes avalanche multiplication, generating an avalanche current. The operation modes when a reverse bias voltage is applied to the APD include Geiger mode and linear mode. The Geiger mode is an operation mode in which a reverse bias voltage greater than the breakdown voltage of the APD is applied between the anode and cathode. The linear mode is an operation mode in which a reverse bias voltage applied between the anode and cathode is near or below the breakdown voltage of the APD. An APD operated in Geiger mode is called a SPAD (Single Photon Avalanche Diode). In this embodiment, the APD constituting the photoelectric conversion element PD is operated in Geiger mode. However, the present invention is not limited to this mode, and the APD may be operated in linear mode.

[0024] The switch unit MP has the role of switching between a period during which photons incident on the photoelectric conversion element PD are detected (photon detection period) and a period during which the photoelectric conversion element PD is charged (charging period). That is, when the switch unit MP receives a low-level signal CLKB from the control unit 60 and turns on, the voltage VH is supplied to node A and the photoelectric conversion element PD is charged. When the switch unit MP receives a high-level signal CLKB from the control unit and turns off, the node A is disconnected from the voltage VH, and the potential of node A becomes variable in response to the incidence of photons.

[0025] The waveform shaping unit 16 shapes the voltage waveform of node A and converts it into a photon detection pulse signal indicating that a photon has entered the photoelectric conversion element PD during the photon detection period. That is, the waveform shaping unit 16 outputs a high-level signal (photon detection signal) when the potential of node A falls below a predetermined determination threshold in response to the incidence of a photon on the photoelectric conversion element PD during the period (photon detection period) when signal CLKB is high. The signal level of the output node (node ​​B) of the waveform shaping unit 16 becomes low at the end of the photon detection period, i.e., at the falling edge when signal CLKB next transitions from high to low. The waveform shaping unit 16 can be configured with a logic circuit such as an inverter circuit.

[0026] The counter 18 is, for example, a counter that counts up each time it detects a rising edge of the photon detection signal output from the waveform shaping unit 16, counts the rising edges of the photon detection signal, and holds a count value that is the counting result. In this way, the counter 18 counts the number of avalanche multiplications that occur during a predetermined exposure period. The count value in the counter 18 is reset in response to a reset signal RES from the control unit 60.

[0027] The counter 18 is connected to the calculation unit C corresponding to the pixel block including the pixel P, without going through a selection circuit or the like. In other words, the calculation unit C can constantly read out the count value held by the counter 18. That is, even if the counter 18 is currently performing a series of counting operations, the calculation unit C can read out the current count value from the counter 18 at any timing. After reading out the count value, the counter 18 continues its counting operation from the current count value.

[0028] Note that the pixel P is not limited to the above-described configuration example, as long as it includes at least a photoelectric conversion unit that outputs a photon detection signal in response to incident photons and a counter that counts the photon detection signal output from the photoelectric conversion unit. For example, the pixel P may be configured to include an active quench circuit instead of the switch unit MP. Furthermore, although the present specification describes the pixel P as including one photoelectric conversion element PD and one counter 18, the present specification is not limited to this example. That is, one pixel P may include multiple photoelectric conversion elements PD and one counter 18 (i.e., one counter 18 is shared by multiple photoelectric conversion elements PD). Furthermore, one pixel P may include one photoelectric conversion element PD and multiple counters 18 (i.e., multiple counters 18 share one photoelectric conversion element PD). The calculation unit C will be described as receiving the count values ​​of the counters 18 of multiple pixels P, but the calculation unit C may be configured to receive the count values ​​of multiple counters 18, each of which counts the photon detection signals of multiple photoelectric conversion elements PD.

[0029] Next, an example configuration of the calculation unit C will be described with reference to FIG. 4. Each calculation unit C may be configured, for example, as shown in FIG. 4, with a count value holding unit 20, a selector 22, a gradient calculation unit 24, a vector calculation unit 26, and an output selection unit 32. The count value holding unit 20 is connected to the selector 22 and the output selection unit 32. The selector 22 is connected to the gradient calculation unit 24. The gradient calculation unit 24 is connected to the vector calculation unit 26. The vector calculation unit 26 is connected to the output selection unit 32. The output selection units 32 of calculation units C adjacent in the column direction are connected to each other. For example, the output selection unit 32 of calculation unit C(m',n') is connected to the output selection unit 32 of calculation unit C(m',n'-4) and the output selection unit 32 of calculation unit C(m',n'+4).

[0030] The calculation unit C(m',n') receives a count value s(x+i,y+j,t) of pixel P included in the corresponding pixel block at time t. Here, x and y represent the column and row numbers corresponding to m', n', i is an integer greater than or equal to 0 and less than I, and j is an integer greater than or equal to 0 and less than J. For example, if the calculation unit C(m',n') corresponds to a 4-column by 4-row pixel block, a total of 16 count values ​​s(x+i,y+j,t) including s(x,y,t), s(x+1,y,t), ..., s(x+3,y+3,t) are input to the calculation unit C(m',n'). The count values ​​s(x+i,y+j,t) are input to the count value holding unit 20, the selector 22, and the output selection unit 32.

[0031] The count value holding unit 20 periodically acquires and holds a count value s(x+i, y+j, t) from pixel P. When the count value holding unit 20 acquires the count value s(x+i, y+j, t) from pixel P, the count value holding unit 20 holds the count value s(x+i, y+j, t-1) from the previous period. The count value holding unit 20 outputs the held count value s(x+i, y+j, t-1) from the previous period to the selector 22.

[0032] The selector 22 sequentially selects two count values ​​s from the count value s(x+i, y+j, t) held by the pixel P and the count value s(x+i, y+j, t-1) from one cycle ago held by the count value holding unit 20, and outputs them to the gradient calculation unit 24.

[0033] The gradient calculation unit 24 calculates gradients in the x direction (row direction), y direction (column direction), and t direction (time direction) for each 2-column by 2-row sub-pixel block SB. A gradient is a parameter that represents spatial and temporal luminance gradients. These gradients can be calculated by calculating the difference between the count values ​​s selected by the selector 22 and then adding this difference to other differences. For example, if a pixel block corresponding to one calculation unit C is composed of 4-column by 4-row pixels P, each pixel block may include four sub-pixel blocks SB1, SB2, SB3, and SB4 arranged in 2 columns and 2 rows, as shown in FIG. 5 . If the gradients in the x direction, y direction, and t direction of sub-pixel block SBk are denoted as grad_xk, grad_yk, and grad_tk, respectively, they can be expressed as follows: grad_x1=s(x,y,t)-s(x+1,y,t) +s(x,y+1,t)-s(x+1,y+1,t) grad_x2=s(x+2,y,t)-s(x+3,y,t) +s(x+2,y+1,t)-s(x+3,y+1,t) grad_x3=s(x,y+2,t)-s(x+1,y+2,t) +s(x,y+3,t)-s(x+1,y+3,t) grad_x4=s(x+2,y+2,t)-s(x+3,y+2,t) +s(x+2,y+3,t)-s(x+3,y+3,t) grad_y1=s(x,y,t)-s(x,y+1,t) +s(x+1,y,t)-s(x+1,y+1,t) grad_y2=s(x+2,y,t)-s(x+2,y+1,t) +s(x+3,y,t)-s(x+3,y+1,t) grad_y3=s(x,y+2,t)-s(x,y+3,t) +s(x+1,y+2,t)-s(x+1,y+3,t) grad_y4=s(x+2,y+2,t)-s(x+2,y+3,t) +s(x+3,y+2,t)-s(x+3,y+3,t) grad_t1=s(x,y,t)-s(x,y,t-1) +s(x+1,y,t)-s(x+1,y,t-1) +s(x,y+1,t)-s(x,y+1,t-1) +s(x+1,y+1,t)-s(x+1,y+1,t-1) grad_t2=s(x+2,y,t)-s(x+2,y,t-1) +s(x+3,y,t)-s(x+3,y,t-1) +s(x+2,y+1,t)-s(x+2,y+1,t-1) +s(x+3,y+1,t)-s(x+3,y+1,t-1) grad_t3=s(x,y+2,t)-s(x,y+2,t-1) +s(x+1,y+2,t)-s(x+1,y+2,t-1) +s(x,y+3,t)-s(x,y+3,t-1) +s(x+1,y+3,t)-s(x+1,y+3,t-1) grad_t4=s(x+2,y+2,t)-s(x+2,y+2,t-1) +s(x+3,y+2,t)-s(x+3,y+2,t-1) +s(x+2,y+3,t)-s(x+2,y+3,t-1) +s(x+3,y+3,t)-s(x+3,y+3,t-1),

[0034] Here, the vector value v of the optical flow to be calculated is v=(v ｘ , v ｙ According to the Lucas-Kanade method, the following optical flow constraint equation holds for each sub-pixel block SB: grad_x1×v ｘ +grad_y1×v ｙ =-grad_t1...(1) grad_x2×v ｘ +grad_y2×v ｙ =-grad_t2...(2) grad_x3×v ｘ +grad_y3×v ｙ =-grad_t3...(3) grad_x4×v ｘ +grad_y4×v ｙ =-grad_t4...(4)

[0035] Here, the following matrix A and matrix b are defined.

[0036] When equations (1) to (4) are rewritten using matrix A and matrix b, they can be expressed as equation (5) below.

[0037] When equation (2) is solved for the vector value v, the following equation (6) is obtained.

[0038] The vector calculation unit 26 calculates the vector value v of the optical flow from the gradient calculated by the gradient calculation unit 24 based on the determinant of equation (6), and outputs the vector value v to the output selection unit 32 .

[0039] In this way, the count value s(x+i, y+j, t) at time t and the vector value v of the optical flow are input to the output selection unit 32. Under the control of the control unit 60, the output selection unit 32 of the calculation unit C(m', n') outputs at least one of the count value s(x+i, y+j, t) and the vector value v to the output selection unit 32 of the calculation unit C(m', n'+4) adjacent to it on the downstream side in the column direction. In this way, by sequentially transferring the data held by the output selection unit 32 of each calculation unit C to the calculation unit C further downstream (on the signal processing unit 40 side), it is possible to output the data of all calculation units C(m', 0) to C(m', N-4) to the signal processing unit 40.

[0040] With this configuration, it is possible to read both the optical flow vector value v and the count value s(x+i, y+j, t). It is also possible to selectively read either the optical flow vector value v or the count value s(x+i, y+j, t). Which one is read can be switched according to an instruction from the control unit 60 depending on the operation mode.

[0041] Next, an example of the operation of the photoelectric conversion device according to this embodiment will be described with reference to Fig. 6. Fig. 6 schematically shows the change over time in the count value of the counter 18 and the operation of the calculation unit C during two consecutive exposure periods.

[0042] The counter 18 continues to perform a series of counting operations during each exposure period. That is, except for resetting the count value at the start of an exposure period, the count value is not reset again until the end of that exposure period. The count value holding unit 20 periodically acquires the count value of the counter 18 multiple times during the exposure period. In the operation example of FIG. 6 , including the time when the counter 18 is reset, the count value is acquired seven times during one exposure period. The calculation unit C of the photoelectric conversion device according to this embodiment can refer to the value of the counter 18 of the pixel P at any time, and can therefore calculate the optical flow by referring to the count value during the exposure period.

[0043] 6, the optical flow is calculated and read five times during one exposure period. After the exposure period has elapsed, the output selection unit 32 of the calculation unit C outputs the count value of the counter 18 at that time. In this operation example, if the speed at which the count value is read (i.e., normal image reading) is 60 fps, for example, the optical flow can be read at a speed equivalent to 300 fps.

[0044] Since optical flow is based on a first-order approximation of a Taylor expansion of a relational expression that represents the movement of an object, the higher the time resolution, the smaller the error in the first-order approximation. In this embodiment, one vector value v is calculated for a pixel block having pixels P of I columns by J rows, so the spatial resolution is 1 / 16 of the data read from the counter 18. Therefore, optical flow can be calculated at high speed without significantly straining the readout bandwidth or the bandwidth of the output IF.

[0045] As described above, according to this embodiment, it is possible to realize a photoelectric conversion device that can acquire optical flow at high speed without requiring data processing by an external device.

[0046] [Second Embodiment] A photoelectric conversion device and a driving method thereof according to a second embodiment of the present invention will be described with reference to Fig. 7. Components similar to those in the photoelectric conversion device according to the first embodiment are given the same reference numerals, and descriptions thereof will be omitted or simplified. Fig. 7 is a block diagram showing an example of the configuration of a calculation unit in the photoelectric conversion device according to this embodiment.

[0047] The photoelectric conversion device according to this embodiment is similar to the photoelectric conversion device according to the first embodiment, except for the configuration of the calculation unit C. In this embodiment, differences from the photoelectric conversion device of the first embodiment will be mainly described, and descriptions of parts that are similar to the photoelectric conversion device of the first embodiment will be omitted as appropriate.

[0048] 7, the photoelectric conversion device according to this embodiment differs from the photoelectric conversion device according to the first embodiment in that the calculation unit C further includes a filter unit 28. The filter unit 28 is connected between the vector calculation unit 26 and the output selection unit 32. The filter unit 28 performs a filter process using, for example, similarity in the time direction on the vector value v output from the vector calculation unit 26, and outputs the processed vector value v' to the output selection unit 32. The filter unit 28 can be implemented as a low-pass filter such as a moving average filter, a Kalman filter, or the like.

[0049] Generally, the count value s(x+i, y+j, t) output from pixel P contains noise components such as optical shot noise, and the vector value v calculated based on the count value s(x+i, y+j, t) also contains error components. By filtering the vector value v in the filter unit 28 and removing the error components superimposed on the vector value v, it is possible to calculate the optical flow with higher accuracy.

[0050] As described above, according to this embodiment, it is possible to realize a photoelectric conversion device that can acquire optical flow at high speed without requiring data processing by an external device.

[0051] [Third Embodiment] A photoelectric conversion device and a driving method thereof according to a third embodiment of the present invention will be described with reference to Fig. 8. Components similar to those of the photoelectric conversion device according to the first or second embodiment are given the same reference numerals, and descriptions thereof will be omitted or simplified. Fig. 8 is a block diagram showing an example of the configuration of a calculation unit in the photoelectric conversion device according to this embodiment.

[0052] The photoelectric conversion device according to this embodiment is similar to the photoelectric conversion device according to the first embodiment, except for the configuration of the calculation unit C. In this embodiment, differences from the photoelectric conversion device of the first embodiment will be mainly described, and descriptions of parts that are similar to the photoelectric conversion device of the first embodiment will be omitted as appropriate.

[0053] 8, the photoelectric conversion device according to this embodiment differs from the photoelectric conversion device according to the first embodiment in that the calculation unit C further includes a motion determination unit 30. The motion determination unit 30 is connected between the vector calculation unit 26 and the output selection unit 32. The motion determination unit 30 has a role of determining whether the calculated optical flow is zero or close to zero, based on the vector value v calculated by the vector calculation unit 26.

[0054] In the first embodiment, regardless of the magnitude of the vector value v of the calculated optical flow, the vector value v is output from all calculation units C. However, depending on the shooting conditions, such as when the camera is fixed, there may be cases where the area where the subject is moving is limited to only a part of the image. In such cases, by outputting the vector value v only from the calculation unit C corresponding to the area where the subject is moving, it is possible to improve the frame rate and reduce power consumption.

[0055] Therefore, in this embodiment, the motion determination unit 30 is configured to determine whether the calculated optical flow is zero or close to zero, and if it is determined that there is no subject movement, it does not output the vector value v, but only outputs a signal F indicating that there is no subject movement. Specifically, if the absolute value of the vector value v is greater than a predetermined value, the motion determination unit 30 determines that there is movement in the subject, and outputs the vector value v received from the vector calculation unit 26 to the output selection unit 32. On the other hand, if the absolute value of the vector value v is equal to or less than the predetermined value, the motion determination unit 30 determines that there is no movement in the subject, and outputs a signal F indicating that there is no subject movement to the output selection unit 32.

[0056] The motion determination unit 30 is not limited to a configuration that determines the presence or absence of motion from the magnitude of the vector value v, but may be configured to determine the presence or absence of motion based on whether the value of the gradient in the t direction (grad_t1 to grad_t4) is close to 0. When determining the presence or absence of motion based on the gradient in the t direction, there is no need to perform calculations in the vector calculation unit 26 if there is no motion, which enables faster processing and lower power consumption.

[0057] As described above, according to this embodiment, it is possible to realize a photoelectric conversion device that can acquire optical flow at high speed without requiring data processing by an external device.

[0058] [Fourth Embodiment] A photoelectric conversion system according to a fourth embodiment of the present invention will be described with reference to Fig. 9. Fig. 9 is a block diagram showing a schematic configuration of the photoelectric conversion system according to this embodiment.

[0059] The photoelectric conversion device 100 described in the first to third embodiments can be applied to various photoelectric conversion systems. Examples of applicable photoelectric conversion systems include digital still cameras, digital camcorders, surveillance cameras, copiers, fax machines, mobile phones, vehicle-mounted cameras, and observation satellites. Camera modules equipped with an optical system such as a lens and an imaging device are also included in photoelectric conversion systems. Figure 9 illustrates a block diagram of a digital still camera as an example of such systems.

[0060] 9 includes an imaging device 201, a lens 202 that forms an optical image of a subject on the imaging device 201, an aperture 204 that adjusts the amount of light passing through the lens 202, and a barrier 206 that protects the lens 202. The lens 202 and the aperture 204 form an optical system that focuses light on the imaging device 201. The imaging device 201 is the photoelectric conversion device 100 described in any one of the first to third embodiments, and converts the optical image formed by the lens 202 into image data.

[0061] The photoelectric conversion system 200 also includes a signal processing unit 208 that processes an output signal output from the imaging device 201. The signal processing unit 208 generates image data from the digital signal output by the imaging device 201. The signal processing unit 208 also performs various corrections and compressions as necessary and outputs the image data. The imaging device 201 may include an AD conversion unit that generates a digital signal to be processed by the signal processing unit 208. The AD conversion unit may be formed in a semiconductor layer (semiconductor substrate) on which the photoelectric conversion unit of the imaging device 201 is formed, or may be formed in a semiconductor layer (semiconductor substrate) different from the semiconductor layer (semiconductor substrate) on which the photoelectric conversion unit of the imaging device 201 is formed. The signal processing unit 208 may also be formed in the same semiconductor layer (semiconductor substrate) as the imaging device 201.

[0062] The photoelectric conversion system 200 further includes a memory unit 210 for temporarily storing image data, and an external interface unit (external I / F unit) 212 for communicating with an external computer or the like. The photoelectric conversion system 200 further includes a recording medium 214 such as a semiconductor memory for recording or reading out image data, and a recording medium control interface unit (recording medium control I / F unit) 216 for recording or reading out data from the recording medium 214. The recording medium 214 may be built into the photoelectric conversion system 200 or may be detachable.

[0063] Furthermore, the photoelectric conversion system 200 has an overall control / calculation unit 218 that performs various calculations and controls the entire digital still camera, and a timing generation unit 220 that outputs various timing signals to the image capture device 201 and the signal processing unit 208. Here, timing signals and the like may be input from outside, and the photoelectric conversion system 200 only needs to have at least the image capture device 201 and the signal processing unit 208 that processes the output signal output from the image capture device 201.

[0064] The imaging device 201 outputs an imaging signal to the signal processing unit 208. The signal processing unit 208 performs predetermined signal processing on the imaging signal output from the imaging device 201 and outputs image data. The signal processing unit 208 generates an image using the imaging signal.

[0065] As described above, according to this embodiment, a photoelectric conversion system can be realized to which the photoelectric conversion device 100 according to the first to third embodiments is applied.

[0066] [Fifth Embodiment] A photoelectric conversion system and a mobile object according to a fifth embodiment of the present invention will be described with reference to Fig. 10A and Fig. 10B. Fig. 10A is a diagram showing the configuration of a photoelectric conversion system according to this embodiment. Fig. 10B is a diagram showing the configuration of a mobile object according to this embodiment.

[0067] FIG. 10A illustrates an example of a photoelectric conversion system for an in-vehicle camera. The photoelectric conversion system 300 includes an image capture device 310. The image capture device 310 is the photoelectric conversion device 100 described in any one of the first to third embodiments. The photoelectric conversion system 300 includes an image processing unit 312 that performs image processing on multiple pieces of image data acquired by the image capture device 310, and a parallax acquisition unit 314 that calculates parallax (phase difference between parallax images) from the multiple pieces of image data acquired by the photoelectric conversion system 300. The photoelectric conversion system 300 also includes a distance acquisition unit 316 that calculates the distance to an object based on the calculated parallax, and a collision determination unit 318 that determines whether or not there is a possibility of a collision based on the calculated distance. Here, the parallax acquisition unit 314 and the distance acquisition unit 316 are examples of distance information acquisition means that acquire information about the distance to the object. In other words, the distance information includes information about the parallax, the defocus amount, the distance to the object, etc. The collision determination unit 318 may use any of this distance information to determine the possibility of a collision. The distance information acquisition means may be realized by dedicated hardware, a software module, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a combination thereof.

[0068] The photoelectric conversion system 300 is connected to a vehicle information acquisition device 320 and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The photoelectric conversion system 300 is also connected to a control ECU 330, which is a control device that outputs a control signal to generate a braking force for the vehicle based on the determination result of the collision determination unit 318. The photoelectric conversion system 300 is also connected to an alarm device 340 that issues an alarm to the driver based on the determination result of the collision determination unit 318. For example, if the determination result of the collision determination unit 318 indicates a high collision possibility, the control ECU 330 performs vehicle control to avoid the collision and mitigate damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 340 warns the user by sounding an alarm, displaying alarm information on a screen such as a car navigation system, or vibrating the seat belt or steering wheel.

[0069] In this embodiment, the photoelectric conversion system 300 captures an image of the surroundings of the vehicle, for example, the front or rear. Fig. 10B shows the photoelectric conversion system when capturing an image of the area in front of the vehicle (image capture range 350). The vehicle information acquisition device 320 sends instructions to the photoelectric conversion system 300 or the image capture device 310. This configuration can further improve the accuracy of distance measurement.

[0070] Although the above describes an example of control to prevent collision with other vehicles, the present invention can also be applied to control of automatic driving by following other vehicles, control of automatic driving to prevent deviation from a lane, etc. Furthermore, the photoelectric conversion system is not limited to vehicles such as the subject vehicle, but can be applied to moving bodies (moving devices) such as ships, aircraft, and industrial robots. In addition, the present invention can be applied not only to moving bodies but also to a wide range of devices that use object recognition, such as intelligent transport systems (ITS).

[0071] Sixth Embodiment An apparatus according to a sixth embodiment of the present invention will be described with reference to Fig. 11. Fig. 11 is a block diagram showing a schematic configuration of the apparatus according to this embodiment.

[0072] FIG. 11 is a schematic diagram showing equipment EQP including a photoelectric conversion device APR. The photoelectric conversion device APR has the functions of the photoelectric conversion device 100 of any one of the first to third embodiments. All or part of the photoelectric conversion device APR is a semiconductor device IC. The photoelectric conversion device APR of this example can be used, for example, as an image sensor, an AF (Auto Focus) sensor, a photometry sensor, or a distance measurement sensor. The semiconductor device IC has a pixel area PX in which pixel circuits PXC including photoelectric conversion units are arranged in a matrix. The semiconductor device IC can have a peripheral area PR around the pixel area PX. Circuits other than pixel circuits can be arranged in the peripheral area PR.

[0073] The photoelectric conversion device APR may have a stacked structure (chip stacked structure) in which a first semiconductor chip provided with a plurality of photoelectric conversion units and a second semiconductor chip provided with peripheral circuits are stacked. The peripheral circuits in the second semiconductor chip may be column circuits corresponding to the pixel columns of the first semiconductor chip. The peripheral circuits in the second semiconductor chip may also be matrix circuits corresponding to the pixels or pixel blocks of the first semiconductor chip. The first and second semiconductor chips may be connected by through-silicon vias (TSVs), inter-chip wiring formed by direct bonding of a conductor such as copper, connection by microbumps between chips, connection by wire bonding, or the like.

[0074] The photoelectric conversion device APR may include, in addition to the semiconductor device IC, a package PKG that houses the semiconductor device IC. The package PKG may include a base to which the semiconductor device IC is fixed, a lid such as glass that faces the semiconductor device IC, and connecting members such as bonding wires and bumps that connect terminals provided on the base to terminals provided on the semiconductor device IC.

[0075] The equipment EQP may further include at least one of an optical device OPT, a control device CTRL, a processing device PRCS, a display device DSPL, a memory device MMRY, and a mechanical device MCHN. The optical device OPT corresponds to the photoelectric conversion device APR as a photoelectric conversion device, and is, for example, a lens, a shutter, or a mirror. The control device CTRL controls the photoelectric conversion device APR and is, for example, a semiconductor device such as an ASIC. The processing device PRCS processes signals output from the photoelectric conversion device APR and constitutes an AFE (analog front end) or a DFE (digital front end). The processing device PRCS is a semiconductor device such as a CPU (central processing unit) or an ASIC (application-specific integrated circuit). The display device DSPL is an EL display device or a liquid crystal display device that displays information (images) obtained by the photoelectric conversion device APR. The memory device MMRY is a magnetic device or a semiconductor device that stores information (images) obtained by the photoelectric conversion device APR. The memory device MMRY is a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive. The mechanical device MCHN has a moving part or a propulsion part such as a motor or an engine. The device EQP displays a signal output from the photoelectric conversion device APR on a display device DSPL or transmits the signal to the outside via a communication device (not shown) provided in the device EQP. For this purpose, the device EQP preferably further includes a memory device MMRY and a processing device PRCS in addition to the memory circuit unit and arithmetic circuit unit provided in the photoelectric conversion device APR.

[0076] The equipment EQP shown in FIG. 11 may be an electronic device such as an information terminal with a photographing function (e.g., a smartphone or a wearable device) or a camera (e.g., an interchangeable lens camera, a compact camera, a video camera, or a surveillance camera). The mechanical device MCHN in the camera can drive components of the optical device OPT for zooming, focusing, and shutter operation. The equipment EQP may also be transportation equipment (mobile object) such as a vehicle, a ship, or an aircraft. The equipment EQP may also be medical equipment such as an endoscope or a CT scanner.

[0077] The mechanical device MCHN in the transportation equipment can be used as a moving device. The equipment EQP as transportation equipment is suitable for transporting the photoelectric conversion device APR and for assisting and / or automating driving (piloting) using a photographing function. The processing device PRCS for assisting and / or automating driving (piloting) can perform processing to operate the mechanical device MCHN as a moving device based on information obtained by the photoelectric conversion device APR.

[0078] The photoelectric conversion device APR according to this embodiment can provide high value to its designer, manufacturer, seller, purchaser, and / or user. Therefore, if the photoelectric conversion device APR is installed in an equipment EQP, the value of the equipment EQP can also be increased. Therefore, when manufacturing and selling equipment EQP, deciding to install the photoelectric conversion device APR according to this embodiment in the equipment EQP is advantageous in terms of increasing the value of the equipment EQP.

[0079] [Modified Embodiments] The present invention is not limited to the above-described embodiments and can be modified in various ways.

[0080] For example, an example in which part of the configuration of any one of the embodiments is added to another embodiment, or an example in which part of the configuration of another embodiment is substituted therefor, is also an embodiment of the present invention.

[0081] In the above embodiment, an example was shown in which optical flow was calculated for each pixel block of 4 columns x 4 rows. However, the pixel block used as the unit for calculating optical flow is not limited to 4 columns x 4 rows and can be set to any size. In the above embodiment, an example was shown in which gradients were calculated for each pixel block of 2 columns x 2 rows. However, the pixel block used as the unit for calculating gradients is not limited to 2 columns x 2 rows and can be set to any size. Furthermore, a Sobel filter, a Prewitt filter, or the like may be used to calculate the gradients. If the photoelectric conversion device is equipped with color filters, the gradients may be calculated independently for each channel. That is, the gradients of R, G, and B may be calculated separately.

[0082] In the above embodiment, data is read by transferring it in the column direction (vertical direction), but it may be configured to read it by transferring it in the row direction (horizontal direction).Furthermore, it may be configured to selectively use column-direction transfer and row-direction transfer for reading count values ​​and vector values.

[0083] Furthermore, the imaging systems shown in the fourth and fifth embodiments above are examples of imaging systems to which the photoelectric conversion device of the present invention can be applied, and imaging systems to which the photoelectric conversion device of the present invention can be applied are not limited to the configurations shown in Figures 9 and 10A.

[0084] The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0085] It should be noted that the above-described embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be carried out in various forms without departing from its technical concept or main features.

[0086] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention.

[0087] This application claims priority based on Japanese Patent Application No. 2023-211216, filed on December 14, 2023, the entire contents of which are incorporated herein by reference.

[0088] C... Calculation section P... Pixel 10... Pixel / Calculation section 20... Count value holding section 22... Selector 24... Gradient calculation section 26... Vector calculation section 28... Filter section 30... Motion determination section 32... Output selection section 40... Signal processing section 50... Output IF section 60... Control section 100... Photoelectric conversion device

Claims

1. A photoelectric conversion device comprising: a plurality of photoelectric conversion units, each of which outputs a photon detection signal in response to the incidence of a photon; a plurality of counters, each of which counts the photon detection signals of a corresponding one of the photoelectric conversion units; and a calculation unit which performs calculation processing based on the count values ​​of each of the plurality of counters, wherein the calculation unit comprises: a holding unit which holds the count values ​​of the plurality of counters; and a vector calculation unit which calculates a vector value representing an optical flow between the second time and the first time based on the count value of the counter at a first time and the count value of the counter at a second time earlier than the first time, which count value is held by the holding unit.

2. The photoelectric conversion device according to claim 1, further comprising an output selection section that selects and outputs at least one of the count value of the counter and the vector value.

3. A photoelectric conversion device as described in claim 2, characterized in that it has the photoelectric conversion unit and a plurality of pixels each having the counter, the plurality of pixels being divided into a plurality of pixel blocks each including two or more of the pixels, the calculation unit being provided for each of the pixel blocks, and each of the calculation units calculating the vector value based on the count value of the pixel included in the corresponding pixel block at the first time and the count value of the pixel included in the corresponding pixel block at the second time.

4. A photoelectric conversion device as described in claim 3, further comprising a signal processing unit which performs signal processing on the count value or the vector value output from the output selection unit, the multiple pixel blocks are arranged in multiple columns, adjacent output selection units arranged in the same pixel block column are connected to each other, and the count value or the vector value output from one output selection unit is transferred to the signal processing unit via another output selection unit arranged in the same pixel block column.

5. A photoelectric conversion device as described in claim 3, further comprising a signal processing unit which performs signal processing on the count value or the vector value output from the output selection unit, the plurality of pixel blocks being arranged in a plurality of rows, adjacent output selection units arranged in the same pixel block row being connected to each other, and the count value or the vector value output from one output selection unit being transferred to the signal processing unit via another output selection unit arranged in the same pixel block row.

6. A photoelectric conversion device as claimed in any one of claims 3 to 5, characterized in that the calculation unit further has a motion determination unit which determines the motion of the subject based on the vector value, and when the motion determination unit determines that there is no motion in the subject, the output selection unit outputs a signal indicating that fact instead of outputting the vector value.

7. A photoelectric conversion device according to any one of claims 3 to 6, characterized in that the pixel is configured so that the count value of the counter can be read out at any time by the calculation unit.

8. A photoelectric conversion device according to any one of claims 1 to 7, characterized in that the calculation unit further has a gradient calculation unit that calculates a gradient representing a spatial and temporal luminance gradient based on the count value at the second time and the count value at the first time, and the vector calculation unit calculates the vector value based on the gradient calculated by the gradient calculation unit.

9. The photoelectric conversion device according to any one of claims 1 to 8, characterized in that the calculation section further comprises a filter section that performs a filter process on the vector values ​​using similarity in the time direction.

10. A photoelectric conversion device according to any one of claims 1 to 9, characterized in that the first time and the second time are included in one exposure period.

11. The photoelectric conversion device according to claim 10, wherein the counter continuously performs a counting operation during the exposure period.

12. A photoelectric conversion device as described in any one of claims 2 to 7, characterized in that the first time and the second time are included in one exposure period, and the output selection unit outputs the count value of the counter when the exposure period has elapsed.

13. The photoelectric conversion device according to any one of claims 10 to 12, characterized in that the calculation unit calculates the vector value a plurality of times during one exposure period.

14. The photoelectric conversion device according to any one of claims 1 to 13, characterized in that the photoelectric conversion section has an avalanche photodiode and outputs a pulse signal that is the photon detection signal in response to incidence of a photon.

15. A photoelectric conversion device according to any one of claims 1 to 14, characterized in that it comprises: a first substrate on which the plurality of pixels are provided; and a second substrate stacked on the first substrate and on which the calculation unit is provided.

16. A photoelectric conversion device comprising: a plurality of photoelectric conversion units, each of which outputs a photon detection signal in response to the incidence of a photon; a plurality of counters, each of which counts the photon detection signals output from a corresponding one of the photoelectric conversion units; and a calculation unit that performs calculation processing based on the count values ​​of the plurality of counters, wherein the calculation unit periodically obtains count values ​​from the plurality of counters while the counters are performing a series of counting operations, outputs a vector value representing an optical flow at a first period based on the obtained count values, and outputs the count values ​​of the counters at a second period longer than the first period.

17. The photoelectric conversion device according to claim 16, wherein the second period corresponds to an exposure period of the photoelectric conversion section.

18. The photoelectric conversion device according to claim 16 or 17, characterized in that each of the plurality of photoelectric conversion sections has an avalanche photodiode.

19. A photoelectric conversion device comprising: a plurality of pixels, each having a photoelectric conversion unit that outputs a photon detection signal in response to incidence of a photon, and a counter that counts the photon detection signal output from the photoelectric conversion unit; and a calculation unit that calculates a vector value representing an optical flow between a first time and a second time based on a count value of the counter at a first time for the plurality of pixels and a count value of the counter at a second time different from the first time, wherein the plurality of pixels are divided into a plurality of pixel blocks, each including two or more of the pixels, and arranged to form a plurality of rows and a plurality of columns, the calculation unit is provided for each of the pixel blocks, adjacent calculation units arranged in the same pixel block row or pixel block column are connected to each other, and the vector value output from one of the calculation units is transferred via another of the calculation units arranged in the same pixel block row or pixel block column.

20. A photoelectric conversion system comprising: a photoelectric conversion device according to any one of claims 1 to 19; and a signal processing device for processing a signal output from the photoelectric conversion device.

21. A moving body comprising: a photoelectric conversion device according to any one of claims 1 to 19; a distance information acquisition means for acquiring distance information to an object from a parallax image based on a signal from the photoelectric conversion device; and a control means for controlling the moving body based on the distance information.

22. An apparatus comprising a photoelectric conversion device according to any one of claims 1 to 19, an optical device corresponding to the photoelectric conversion device, a control device for controlling the photoelectric conversion device, a processing device for processing a signal output from the photoelectric conversion device, a mechanical device controlled based on information obtained by the photoelectric conversion device, a display device for displaying information obtained by the photoelectric conversion device, and at least one of a memory device for storing information obtained by the photoelectric conversion device.

Images