Photoelectric conversion device, photoelectric conversion system, equipment and mobile body

Abstract

To provide a technology advantageous to performance improvement of a photoelectric conversion device.SOLUTION: A photoelectric conversion device includes a plurality of pixels respective of which include photoelectric conversion elements, and a plurality of operation parts. The plurality of pixels and the plurality of operation parts are arranged in two-dimensional arrays respectively. The plurality of pixels are connected to corresponding operation parts of the plurality of operation parts every pixel group composed of two or more pixels of the plurality of pixels. The respective plurality of pixels output spike-like signals to the connected operation parts of the plurality of operation parts. The respective plurality of operation parts perform operation with respect to the spike-like signals.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a photoelectric conversion device, a photoelectric conversion system, an apparatus, and a mobile object. [Background technology]

[0002] With the recent spread of IoT, AI, and autonomous driving, there is a demand for image sensors with lower power consumption and higher speed. Non-Patent Document 1 shows an event-based sensor that monitors changes in the amount of light with each pixel arranged in a two-dimensional array and outputs a signal when a change is detected. According to Non-Patent Document 1, the event-based sensor outputs a signal only when a change in the amount of light occurs, enabling low power consumption and high-speed operation. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Finateu et al., "A 1280x720 Back-Illuminated Stacked Temporal Contrast Event-Based Vision Sensor with 4.86μm Pixels, 1.066GEPS Readout, Programmable Event-Rate Controller and Compressive Data-Formatting Pipeline," 2020 IEEE International Solid-State Circuits Conference, (USA), February 2020, pp.112-114. Summary of the Invention [Problem to be solved by the invention]

[0004] In the configuration of Non-Patent Document 1, when a change in the amount of light is detected, a request signal is output from the transfer circuit for each pixel to the arbitration circuit, and the arbitration circuit selects the row to be read out. However, since the signal is read out row by row, when a request signal is output from pixels in two or more rows, some pixels will wait until the signal is read out. For example, if it takes 1 μs to read out one row, and the sensor is mounted on a moving object or the like and a change in the amount of light is detected in many pixels, if there are 720 rows, a maximum delay time of just under 1 ms will occur. Even if the time resolution of the time stamp added in response to the detection of a change in the amount of light in a pixel is high, such as 1 μs, the time resolution of the signal actually read out will be equivalent to 1 ms. This difference in time resolution can cause artifacts, such as a straight line being observed as a curve. In addition, if the next change in the amount of light occurs in the same pixel before the signal is read out, the previous signal may be lost.

[0005] An object of the present invention is to provide a technique that is advantageous for improving the performance of a photoelectric conversion device. [Means for solving the problem]

[0006] In view of the above problems, a photoelectric conversion device according to an embodiment of the present invention is a photoelectric conversion device comprising a plurality of pixels each including a photoelectric conversion element, and a plurality of calculation units, wherein the plurality of pixels and the plurality of calculation units are each arranged in a two-dimensional array, the plurality of pixels are connected to a corresponding one of the plurality of calculation units for each pixel group consisting of two or more pixels among the plurality of pixels, each of the plurality of pixels outputs a spike-shaped signal to the calculation unit among the plurality of calculation units to which it is connected, and each of the plurality of calculation units performs calculations on the spike-shaped signal. Effect of the Invention

[0007] According to the present invention, it is possible to provide a technique that is advantageous for improving the performance of a photoelectric conversion device. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a photoelectric conversion system including a photoelectric conversion device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a configuration example of the photoelectric conversion device in FIG. 1. [Diagram 3] FIG. 2 is a diagram showing an example of the configuration of a pixel in the photoelectric conversion device of FIG. 1; [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a calculation unit of the photoelectric conversion device of FIG. 1. [Diagram 5] FIG. 2 is a diagram showing an example of the configuration of a pixel in the photoelectric conversion device of FIG. 1; [Figure 6] FIG. 2 is a diagram showing an example of the configuration of a calculation unit of the photoelectric conversion device of FIG. 1. [Figure 7] FIG. 2 is a diagram showing a configuration example of the photoelectric conversion device in FIG. 1. [Figure 8] FIG. 2 is a diagram showing a configuration example of the photoelectric conversion device in FIG. 1. [Figure 9] FIG. 2 is a diagram showing a configuration example of the photoelectric conversion device in FIG. 1. [Figure 10] FIG. 1 is a diagram showing an example of the configuration of a device incorporating a photoelectric conversion device according to an embodiment of the present invention. [Figure 11] FIG. 1 is a diagram showing a configuration example of a moving object in which a photoelectric conversion device according to an embodiment of the present invention is incorporated. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] A photoelectric conversion device and a photoelectric conversion system according to an embodiment of the present disclosure will be described with reference to FIGS. 1 to 8. FIG. 1 is a diagram showing a configuration example of the photoelectric conversion system SYS according to the present embodiment. The photoelectric conversion system SYS includes a photoelectric conversion device 10 and a calculation device 20. The photoelectric conversion device 10 performs a predetermined calculation on signals detected by a plurality of pixels each including a photoelectric conversion element, and acquires a signal representing a feature amount of an image (video) according to light incident on the photoelectric conversion device 10. The calculation device 20 receives a signal representing the feature amount from the photoelectric conversion device 10, and further performs a predetermined calculation. The calculation device 20 performs a recognition process, for example, to obtain the position and optical flow of a target object. Such a photoelectric conversion system SYS may be called a vision system. The photoelectric conversion system SYS may be mounted on an autonomous mobile object such as a drone, and may be used for detecting obstacles and creating a map (SLAM: Simultaneous Localization and Mapping). The photoelectric conversion system SYS may also be used for in-vehicle applications such as automobiles, and may be used for detecting pedestrians and other vehicles. The arithmetic device 20 may be an embedded device such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit). The arithmetic device 20 may also be a PC (Personal Computer) or a cloud (Cloud Computing). The photoelectric conversion device 10 and the arithmetic device 20 do not need to be separated as shown in FIG. 1. For example, the photoelectric conversion device 10 and the arithmetic device 20 may be implemented on the same chip.

[0011] With reference to FIG. 2, a circuit configuration example of the photoelectric conversion device 10 will be described. The photoelectric conversion device 10 includes a control unit 110, a pixel / calculation unit 120, a processing unit 180, and an output IF unit 190. The control unit 110 controls the operation of the pixel / calculation unit 120 and the processing unit 180. The pixel / calculation unit 120 includes a plurality of pixels P each including a photoelectric conversion element and a plurality of calculation units C. As shown in FIG. 2, the plurality of pixels P and the plurality of calculation units C are arranged in a two-dimensional array. The plurality of pixels P and the plurality of calculation units C may be arranged on the same substrate. The photoelectric conversion device 10 may have a laminated structure in which a substrate on which at least the photoelectric conversion elements of the plurality of pixels P are arranged and a substrate on which the plurality of calculation units C are arranged are laminated.

[0012] In the configuration shown in Fig. 2, M x N pixels P are arranged in pixel calculation unit 120. When referring to a specific pixel P among the pixels P, it is represented as pixel P(m, n). m and n are integers satisfying 0 < m < (M-1) and 0 < n < (N-1), respectively. Here, it is assumed that M pixels P are arranged in the row direction (horizontal direction in Fig. 2) and N pixels P are arranged in the column direction (vertical direction in Fig. 2).

[0013] The calculation unit C is arranged for each of a predetermined number of pixels P. It can also be said that the pixels P are connected to a corresponding calculation unit C among the calculation units C for each pixel group consisting of two or more pixels P among the pixels P. For example, in the configuration shown in FIG. 2, one calculation unit C is arranged for a pixel group consisting of 4×4 pixels P. When indicating a specific calculation unit C among the calculation units C, it is indicated as calculation unit C(m', n'). In the case of the configuration shown in FIG. 2, m' and n' are multiples of 4. As shown in FIG. 2, the pixel P is directly connected to the corresponding calculation unit C. In other words, other configurations such as a readout mechanism such as a transfer circuit or an arbitration circuit shown in Non-Patent Document 1 may not be arranged between the pixel P and the calculation unit C.

[0014] The calculation unit C is connected to the pixel P included in the corresponding pixel group, as well as to the calculation units C arranged adjacent to each other, and can exchange data. That is, the calculation unit C(m', n') is also connected to the calculation units C(m'-4, n'), (m', n'-4), (m'+4, n'), and (m', n'+4). The calculation unit C performs a predetermined calculation on the signal input from the pixel P and the signal (data) input from the calculation units C adjacent to each other. In addition, the calculation unit C can send the calculation result to the calculation unit C adjacent to each other. In addition, the calculation unit C(m', N-4) arranged at the end of the pixel calculation unit 120 can output the calculation result to the processing unit 180.

[0015] The processing unit 180 performs processing such as executing a predetermined calculation on the calculation results input from each of the multiple calculation units C. Furthermore, the processing unit 180 outputs the processing results to the outside of the photoelectric conversion device 10, such as the calculation device 20, via the output IF unit 190. In this way, by using a configuration in which the signals of the pixels P are directly output to the calculation unit C for each pixel group, the signals generated in each pixel P can be processed with little delay time without being limited by a readout mechanism such as a transfer circuit or an arbitration circuit as shown in Non-Patent Document 1 (no retention).

[0016] Next, the pixel P will be described. In this embodiment, each of the pixels P detects a change in the amount of incident light and outputs a spike-shaped signal. More specifically, each of the multiple pixels P outputs a spike-shaped signal in response to a change in the signal output from the photoelectric conversion element. Since the pixel P outputs a signal in response to a change (event) in the amount of incident light, this spike-shaped signal can also be called an event signal. Each of the multiple pixels P outputs a spike-shaped signal as an event signal to the connected calculation unit C among the multiple calculation units C, and each of the multiple calculation units C performs calculations on the spike-shaped signal. Here, the spike-shaped signal can be, for example, a pulse-shaped signal. Also, for example, the spike-shaped signal may be a signal having a shape such as a sine wave in which the rising and falling edges of the pulse are observed.

[0017] The pixel P can be configured, for example, by the circuit shown in Fig. 3. In the configuration shown in Fig. 3, the pixel P includes a photodiode 210, a logarithmic I / V conversion circuit 220, a subtraction circuit 230, and a comparison circuit 240. The photodiode 210, which is a photoelectric conversion element, generates a photocurrent I p The logarithmic I / V conversion circuit 220 generates the photocurrent I p is converted to potential and converted to logarithmic potential V l Since the logarithmic conversion is mainly intended to expand the dynamic range, the logarithmic conversion may not be performed. In other words, the photocurrent I generated in the photodiode 210 is converted between the photodiode 210 and the subtraction circuit 230. p The subtraction circuit 230 generates a potential according to the signal (photocurrent I p ) logarithmic potential V according to the signal value l The subtraction value V obtained by subtracting the reference potential (reference value) from d The comparison circuit 240 generates the subtraction value V d The comparison circuit 240 (pixel P) compares the subtraction value V d When exceeds a positive threshold T1 or falls below a negative threshold T2, the pixel P(m,n) outputs an event signal E. The event signal E of the pixel P(m,n) at time t is expressed by equation (1).

[0018]

number

[0019] In the photoelectric conversion device 10, the resolution of time t is, for example, 1 μs. The event signal E has high time resolution because it is generated independently (asynchronously) from a frame synchronization signal as in a normal image sensor. The event signal E is output from the pixel P to the corresponding calculation unit C of each pixel P. The event signal E is also used to update the reference potential. In other words, the logarithmic potential V lis held and used as the next reference potential. With the above configuration, an increase or decrease in the amount of light incident on the photodiode 210, which is a photoelectric conversion element, can be output as a spike-shaped event signal E.

[0020] As described above, the photoelectric conversion device 10 can be realized as a single-layer sensor having one substrate, or as a stacked sensor having two or more substrates. In the case of a stacked type, for example, the photoelectric conversion element, the photodiode 210, the subtraction circuit 230, and the comparison circuit 240 may be arranged on separate substrates. In this case, a part of the logarithmic I / V conversion circuit 220 may be arranged on the substrate on which the photodiode 210 is arranged, and another part of the logarithmic I / V conversion circuit 220 may be arranged on the substrate on which the subtraction circuit 230 and the comparison circuit 240 are arranged. Furthermore, in this case, the calculation unit C may be arranged on the substrate on which the subtraction circuit 230 and the comparison circuit 240 are arranged, or may be arranged on another substrate.

[0021] An example of a stacked sensor in which a plurality of substrates are stacked will be described with reference to FIGS. 9(a) and 9(b). FIG. 9(a) is a diagram showing the configuration of a stacked sensor. A photodiode 210 and various semiconductor regions R11 are provided in a semiconductor layer Sem1. The photodiode 210 receives light transmitted through a microlens ML provided on the back side of the semiconductor layer Sem1 and performs photoelectric conversion. A structure W1 including a transistor gate structure G11, a plurality of wiring layers, and a plurality of interlayer insulating films is provided on the front side of the semiconductor layer Sem1. That is, the stacked sensor shown in FIG. 9(a) is configured as a back-illuminated sensor. Here, the microlens ML is not an essential component. A junction layer Z1 is provided on the lower surface of the structure W1. A connection portion M11 made of a metal is provided on this junction layer Z1. This connection portion M11 is typically formed of one of copper (Cu), aluminum (Al), and tungsten (W), or a combination of a plurality of metals. The connection portion M11 is provided inside the insulating film of the junction layer Z1. The substrate Sub1 includes a semiconductor layer Sem1, a structural body W1, and a bonding layer Z1.

[0022] The semiconductor layer Sem2 is provided with various semiconductor regions R21 that constitute various circuits. A structural body W2 including a transistor gate structure G21, a plurality of wiring layers, and a plurality of interlayer insulating films is provided on the front surface side of the semiconductor layer Sem2. A bonding layer Z2 is provided on the upper surface of the structural body W2. The bonding layer Z2 is provided with a connection part M21 made of a metal. The connection part M21 is typically made of one or a combination of metals selected from copper (Cu), aluminum (Al), and tungsten (W). The connection part M21 is provided inside the insulating film of the bonding layer Z2. The substrate Sub2 includes the semiconductor layer Sem2, the structural body W2, and the bonding layer Z2.

[0023] 9(b) shows the state before the substrates Sub1 and Sub2 are bonded. In the state before bonding, the connection parts M11, M21 provided on each of the substrates Sub1 and Sub2 are connected to the wiring layer inside the corresponding substrate. This wiring layer is connected to the functional elements of the corresponding substrate, such as the transistor, the resistive element, the capacitive element, and the photodiode 210. Therefore, the connection parts M11, M21 of each of the multiple substrates Sub1, Sub2 are connected to the functional elements of the corresponding substrate before the substrates Sub1, Sub2 are bonded to each other.

[0024] The bonding surface P1 of the bonding layer Z1 is provided with an insulating film and a connection portion M11 provided by removing a portion of the insulating film. Similarly, the bonding surface P2 of the bonding layer Z2 is provided with an insulating film and a connection portion M21 provided by removing a portion of the insulating film.

[0025] Thereafter, the bonding surface P1 and the bonding surface P2 are bonded together to bond the substrates Sub1 and Sub2. That is, the bonding between the substrates Sub1 and Sub2 is achieved by bonding the insulating films of the substrates Sub1 and Sub2 together, and by bonding the connection portions of the substrates Sub1 and Sub2 together (the connection portions M11 and M21). In this way, electrical conduction between the substrates Sub1 and Sub2 can be achieved.

[0026] The connection part M11 described here may be provided one by one in correspondence with each of the multiple photodiodes 210. In this case, it is possible to provide one connection part M21 for each of the multiple subtraction circuits 230. In this case, it is also possible to provide one connection part M21 for two or more subtraction circuits 230 of the multiple subtraction circuits 230. In addition, it is also possible to provide one connection part M11 for two or more photodiodes 210 of the multiple photodiodes 210. In this case, it is also possible to provide one connection part M21 for each of the multiple subtraction circuits 230, and it is also possible to provide one connection part M21 for two or more subtraction circuits 230 of the multiple subtraction circuits 230.

[0027] The above-mentioned bonding method is an example, and it is also possible to achieve electrical conduction between the multiple substrates by other bonding methods. For example, a laminate is produced by bonding the multiple substrates together with no connection portion provided on the bonding surfaces of each substrate and an insulating film provided thereon. Then, a first through hole penetrating from the upper surface of the laminate to the wiring layer of one substrate and a second through hole penetrating to the wiring layer of the other substrate are provided. Then, a metal is filled into each of the first through hole and the second through hole. Also, a metal film is provided to connect the metal filled in the first through hole and the metal filled in the second through hole. This metal is typically formed of one of copper (Cu), aluminum (Al), and tungsten (W), or a combination of a plurality of metals. This allows electrical conduction between the multiple substrates.

[0028] Next, the calculation performed by the calculation unit C will be described. In this embodiment, the calculation unit C may include a neural network. Furthermore, in this embodiment, the calculation unit C may be configured to perform at least a part of the calculation of a Convolutional Spiking Neural Network (CSNN). Each of the calculation units C is configured, for example, by a circuit as shown in FIG. 4. In the configuration shown in FIG. 4, the calculation unit C includes an L1 calculation unit 310, an L2 calculation unit 320, and a communication unit 330.

[0029] The CSNN calculation in the L1 calculation unit 310 and the L2 calculation unit 320 may be configured by a combination of a convolution calculation, a leaky integrate and fire (LIF) calculation, and a pooling calculation. An input to the L1 calculation unit 310 is an event signal E output from each pixel P connected to the calculation unit C. For the event signal E input to the L1 calculation unit 310, the L1 calculation unit 310 first performs a convolution calculation with a predetermined kernel size (e.g., 4×4 pixels) and a predetermined stride (e.g., 2 pixels). The convolution calculation is expressed, for example, as in Equation (2).

[0030]

number

[0031] Next, the LIF calculation is performed for each channel on the result of the convolution calculation. In the LIF calculation, for example, the membrane potential V(t) expressed by the formula (3) is calculated.

[0032]

number

[0033] A MAX pooling operation is performed on the result of the LIF operation with a predetermined kernel size (e.g., 2x2 pixels) and a predetermined stride (e.g., 2 pixels). Since the result of the LIF operation is spike-like, the result of the MAX pooling operation is equivalent to a logical sum, and the output is also spike-like. However, if a refractory period is provided, the MAX pooling operation will ignore spike-like signals input from the LIF operation for a certain period after firing.

[0034] As a modified example, the processing order of the LIF calculation and the MAX pooling calculation may be partially interchanged, the MAX pooling calculation may be performed on the membrane potential V(t), and the result may be compared with the firing threshold to output a spike-like signal. Although the pooling calculation is expected to improve robustness against positional deviations and reduce the amount of calculation, it is not essential and the pooling calculation may not be performed.

[0035] With the above configuration, the L1 calculation unit 310 outputs the calculation result S1. The calculation result S1 has a coarser spatial resolution (resolution) than the event signal E due to convolution and pooling operations, for example, 1 / 16. The interval between spike-like signals also becomes wider, for example, the time resolution becomes 1 / 10 of that of the event signal E.

[0036] The calculation result S1 is sent to the L2 calculation unit 320 of the same calculation unit C or to the L2 calculation unit 320 of an adjacent calculation unit C. The L2 calculation unit 320 performs convolution calculation, LIF calculation, and pooling calculation in the same manner as the L1 calculation unit 310 to obtain a spike-like calculation result S2. The spatial resolution and time resolution of the calculation result S2 are coarser than those of the calculation result S1. The calculation result S2 is sent to the communication unit 330.

[0037] The communication unit 330 is connected to the communication units 330 of the calculation units C adjacent to each other in the column direction, and is connected to the processing unit 180 at the lower end of the pixel / calculation unit 120. The calculation result Sc (the calculation result Sc may be the same as the calculation result S2, for example, or data may be added to the calculation result S2) output from the communication unit 330 is relayed and sent to the calculation unit C closer to the processing unit 180 in order. Finally, the calculation result Sc of each calculation unit C is input to the processing unit 180. In this way, each of the multiple calculation units C performs a calculation based on the membrane potential corresponding to the received spike-like event signal E, and is finally sent as the calculation result Sc to the processing unit 180 outside the pixel / calculation unit 120. The processing unit 180 may, for example, obtain a histogram on the calculation result Sc as necessary, or may perform further CSNN calculation.

[0038] With the above configuration, event features such as a sequence of events in the time and space directions can be extracted from an event signal E with high time resolution using CSNN calculations. Using the event features thus obtained, a downstream calculation device 20 performs recognition processing such as object detection. Recognition processing such as object detection may be performed using a trained model that has trained the relationship between the event features and the object positions using, for example, a machine learning method based on a CNN (Convolutional Neural Network). Since it is based on information with high time resolution, it is possible to perform highly accurate object detection even when, for example, a high-speed moving object is the detection target.

[0039] The photoelectric conversion device 10 of this embodiment does not have a concept of a frame period, which is common in ordinary sensors, and the state is updated asynchronously at any time to extract the event feature, so the delay from the change in light amount to the extraction of the feature is short. In addition, the event signal E is usually detected only near the edge of a moving object, so it occurs sparsely. Therefore, convolution calculations and the like only need to be performed when an event occurs, and the photoelectric conversion device 10 having the above-mentioned configuration can keep the average power consumption low. In addition, the event signal E output from the pixel P is directly input to the calculator C without going through a readout mechanism including a transfer circuit and an arbitration circuit as shown in Non-Patent Document 1, so it is processed with low delay and is not subject to rate constraints due to the bandwidth of the readout mechanism. In other words, even if the event signal E occurs frequently, it can be processed without any problems. In addition, the feature calculated and processed by the calculator C and the processor 180 has a coarse spatial resolution and time resolution compared to the event signal E, so the amount of information (number of bytes) can be kept low. Therefore, even if the communication band of the output IF unit 190 is relatively narrow, data can be transmitted to the arithmetic device 20, and the power consumption required for transmission is reduced.

[0040] In this embodiment, the calculation unit C may be implemented using an analog calculator as long as it can obtain equivalent calculation results (including approximate calculation results). The calculation unit C may also be implemented using an asynchronous (clockless) digital calculator, or a clock-synchronous digital calculator. Furthermore, the calculation unit C may be implemented as software using a processor. The calculation unit C may also be a combination of these methods. A calculator such as the one described above may also be shared by multiple calculation units C and used in a time-division manner.

[0041] In addition, the order of calculations may be different from that described above in the pixel P and the calculation unit C. For example, in the pixel P shown in FIG. lsubtracting a reference potential from the thresholds T1 and T2 is equivalent to adding a reference potential to the thresholds T1 and T2. In the L1 calculation unit 310 and the L2 calculation unit 320, the convolution calculation, the LIF calculation, and the pooling calculation may be performed in any order. The number of convolution calculations, the LIF calculations, and the pooling calculations may be any number. In addition, recurrent connections may be included in the CSNN.

[0042] In the above embodiment, an example has been shown in which a calculation is performed by CSNN on an event signal E that asynchronously detects a change in the amount of light incident on the photodiode 210, which is a photoelectric conversion element. However, the configuration of the pixel P is not limited to this. FIG. 5 is a diagram showing an example of a configuration of the pixel P of this embodiment that is different from that shown in FIG. 3. In the configuration shown in FIG. 5, the logarithmic potential V l is input to the VCO (Voltage Controlled Oscillator) 510. The VCO 510 outputs a logarithmic potential V l The VCO 510 generates a spike signal S0, which is a spike-like (sine wave) signal train with a frequency according to the frequency of the signal output from the photodiode 210, which is a photoelectric conversion element. That is, each of the multiple pixels P outputs a spike-like signal with a frequency according to the signal value of the signal output from the photodiode 210, which is a photoelectric conversion element. The VCO 510 can be realized, for example, by using a ring oscillator in which three (an odd number of) inverters are connected in a ring shape. In the case of the VCO 510, the pulse duty is about 50%, but as described above, the calculation unit C does not depend on the pulse width of the spike-like signal, so the signal output from the VCO 510 can be treated equivalently to the above-mentioned event signal E.

[0043] Even if the VCO 510 is used for the pixel P, the processing after the calculation unit C that receives the spike signal S0 may be the same as described above. In this embodiment, a feature based on the absolute value of the photocurrent generated by the photodiode 210 is extracted by the CSNN. By learning the relationship between this feature and the object position, etc. using a CNN-based machine learning method, the latter-stage calculation device 20 can perform recognition processing such as object detection as described above. Also, as in the above embodiment, since asynchronously generated signals are processed by the CSNN, it is possible to utilize information with high time resolution. Also, the configuration of the pixel P including the VCO 510 can detect stationary objects with no change in light amount.

[0044] Furthermore, a SPAD (Single Photon Avalanche Diode) element using an avalanche photodiode as a photoelectric conversion element may be used in pixel P. The SPAD element converts each photon into a spike-like signal by utilizing avalanche multiplication. Since the interval at which the spike-like signal is output may depend on the amount of light, a spike-like signal train similar to that of pixel P using the above-mentioned VCO 510 is generated. This spike-like signal train is input to the above-mentioned calculation unit C and may be processed in the same manner as described above. When a SPAD element is used in pixel P, a photoelectric conversion system SYS capable of recognizing an object even under low illuminance can be realized.

[0045] In each of the above-mentioned embodiments, an example was shown in which a signal input from a pixel P was processed by a CSNN, but the calculation unit C may process the input signal by an ANN (Artificial Neural Network). Unlike a CSNN, an ANN is expressed by a differentiable formula, and therefore has the advantage of being able to perform highly accurate learning using an error backpropagation method or the like. On the other hand, since an asynchronous signal such as the above-mentioned event signal E cannot be handled, synchronization is required. Although it is difficult to detect the event signal E with high time resolution by synchronization, it is possible to perform recognition processing such as object detection with high accuracy using a trained model obtained by high-precision learning depending on the recognition target and application.

[0046] FIG. 6 shows a configuration example of a calculation unit C including an ANN. An event signal E input to the calculation unit C is input from, for example, a pixel P as shown in FIG. 3 described above. The synchronization unit 610 counts the input event signal E every predetermined period (for example, 1 ms). The synchronization unit 610 sends the number Ec of event signals E for each pixel P counted for each period (frame) to the L1 calculation unit 620. The number Ec of event signals E becomes image-like data (frame data) synchronized for each period in the photoelectric conversion device 10. Therefore, the L1 calculation unit 620 and the L2 calculation unit 630 can execute general CANN (Convolutional ANN) calculation. For example, the L1 calculation unit 620 performs convolution calculation, ReLU (Rectfied Linear Unit) calculation, pooling calculation, etc. The L2 calculation unit 630 also performs convolution calculation, ReLU calculation, pooling calculation, etc. The calculation result I1 of the L1 calculation unit 620 and the calculation result I2 of the L2 calculation unit 630 also become image-like data with the same period as the number Ec of event signals E. The communication unit 640, like the above-mentioned communication unit 330, sequentially transmits the calculation results Ic, and the obtained feature amount data is output to the calculation device 20 via the processing unit 180 and the output IF unit 190.

[0047] Next, the arrangement of the pixels P and the calculation unit C will be described. In the configuration shown in FIG. 2, one calculation unit C is arranged for every 4×4 pixels P. In this case, for example, in the case of a single-layer structure in which the pixels P and the calculation unit C are arranged on one substrate, it is necessary to widen the pitch at which the pixels P are arranged by the amount of the calculation unit C arranged, compared with the pixel-calculation unit 120 in which only the pixels P are arranged. Even if the photodiode 210, which is a photoelectric conversion element, and the calculation unit C are arranged on different substrates, if the subtraction circuit 230 and the comparison circuit 240 are arranged on the substrate on which the calculation unit C is arranged, the arrangement of the pixels P and the calculation unit C may be as shown in FIG. 2 in the orthogonal projection on the substrate surface. That is, it may be necessary to widen the pitch at which the pixels P are arranged. That is, this leads to a decrease in the spatial resolution of the photoelectric conversion device 10.

[0048] Therefore, for example, as shown in FIG. 7, in the pixel calculation unit 120, the pixels P arranged in a two-dimensional array may be thinned out at a predetermined interval, and each of the calculation units C may be arranged in the area where the pixels P are thinned out. In addition, when the photoelectric conversion device 10 has a stacked structure including a plurality of substrates, each of the calculation units C may be arranged so as to overlap the area where the pixels are thinned out. FIG. 7 shows an example in which one pixel is missing for every 16 pixels (4×4 pixels). Specifically, pixels P(m+1, n+1) (m and n are multiples of 4) are thinned out, and the calculation unit C is arranged in the area where the pixels P are thinned out. The pixels P connected to each calculation unit C are 15 pixels P excluding the thinned out pixels among the 16 neighboring pixels (connection is not shown). In addition, in each calculation unit C, the calculation units C on the four adjacent sides are connected to each other in the same manner as in the configuration shown in FIG. 2. The number of pixels P that input signals to the calculation unit C is 15, and features can be extracted by the same calculation as described above. In the configuration shown in Fig. 7, the interval at which the pixels P are thinned out coincides with the interval at which the calculation units C are arranged. However, this is not limited to this, and for example, one pixel may be thinned out for every 32 pixels, and two calculation units C may be arranged in the area where the pixels are thinned out. Here, in Fig. 7, the control lines by which the control unit 110 shown in Fig. 2 controls the operations of the pixel / calculation unit 120 and the processing unit 180 are also omitted.

[0049] With the above-described configuration, the effect on the arrangement pitch of the pixels P caused by arranging the calculation unit C in the pixel / calculation unit 120 is reduced. As a result, high spatial resolution is realized in the photoelectric conversion device 10. Also, although the amount of information is reduced by thinning out the pixels P, the information of the area in which the calculation unit C is arranged can be supplemented by information obtained from the pixels P arranged around the calculation unit C, and the effect on the final recognition performance is small.

[0050] FIG. 8 is a diagram for explaining a modified example of the arrangement of the pixel P and the calculation unit C shown in FIG. 7. In the configuration shown in FIG. 7, the light incident on the region where the calculation unit C is arranged may not be used for photoelectric conversion. The configuration shown in FIG. 8 assumes the case where the photoelectric conversion device 10 has a laminated structure including a plurality of substrates. For example, a substrate on which the photodiode 210, which is a photoelectric conversion element, and a part of the logarithmic I / V conversion circuit 220 are arranged, and a substrate on which the other part of the logarithmic I / V conversion circuit 220, the subtraction circuit 230, the comparison circuit 240, and the calculation unit C are arranged are laminated. For each pixel P, the photodiode 210 and a part of the logarithmic I / V conversion circuit 220, the other part of the logarithmic I / V conversion circuit 220, the subtraction circuit 230, and the comparison circuit 240 may be arranged so as to overlap each other. On the other hand, a space may be generated in the region of the substrate on which the photodiode 210 is arranged that overlaps with the calculation unit C. Therefore, for example, as shown in Figure 8, the photoelectric conversion element P(m+2, n+1) (m and n are multiples of 4) of the 4 x 4 pixels extends to an area overlapping with the calculation unit C so as to receive incident light in an area of ​​two pixels.

[0051] That is, the photoelectric conversion element (photodiode 210) arranged in each of the multiple pixels P may include a photoelectric conversion element of normal size and a photoelectric conversion element larger than the normal size photoelectric conversion element. The multiple pixels P are arranged such that pixels having a predetermined number of normal photoelectric conversion elements are arranged between two pixels having large photoelectric conversion elements. At this time, each of the multiple calculation units C is arranged so as to overlap a large photoelectric conversion element among the photoelectric conversion elements arranged in the multiple pixels P. With such a configuration, it is possible to improve the utilization efficiency of light incident on the pixel calculation unit 120 of the photoelectric conversion device 10.

[0052] Photocurrent I generated at pixel P(m+2, n+1) pis approximately twice as much as that of other pixels P when the same amount of light is incident. However, when the signal output from pixel P(m+2, n+1) is used as the above-mentioned event signal E, there is no problem with the calculation unit C treating the signal output from pixel P having a large photoelectric conversion element as equivalent to the signals output from other pixels P. Furthermore, under low illuminance, the quality of the event signal E output from pixel P(m+2, n+1) may be higher than that of other pixels P. Therefore, under low illuminance, the calculation unit C may adjust the weighting of the event signal E output from pixel P(m+2, n+1) to increase the contribution rate of the event signal E output from pixel P(m+2, n+1) in the calculation result Sc.

[0053] Hereinafter, as an application example of the photoelectric conversion device 10 (photoelectric conversion system SYS), an apparatus EQP including the photoelectric conversion device 10 (photoelectric conversion system SYS) shown in FIG. 10 will be described. The apparatus EQP may also be called an electronic device. FIG. 10 shows a camera as an example of the apparatus EQP. Here, the concept of a camera includes not only an apparatus whose main purpose is to take pictures, but also an apparatus that auxiliary has a photography function (for example, a personal computer or a mobile terminal such as a smartphone).

[0054] The photoelectric conversion device 10 may be a semiconductor chip with a stacked structure in which a pixel / arithmetic unit 120 is provided. The photoelectric conversion device 10 is accommodated in a semiconductor package PKG as shown in FIG. 10. The package PKG may include a base on which the photoelectric conversion device 10 is fixed, a cover such as glass facing the photoelectric conversion device 10, and a conductive connection member such as a bonding wire or bump that connects a terminal provided on the base to a terminal provided on the photoelectric conversion device 10. The equipment EQP may further include at least one of an optical system OPT, a control device CTRL, a processing device PRCS, a display device DSPL, and a memory device MMRY.

[0055] The optical system OPT forms an image on the photoelectric conversion device 10, and may be, for example, a lens, a shutter, or a mirror. The control device CTRL controls the operation of the photoelectric conversion device 10, and may be, for example, a semiconductor device such as an ASIC. The processing device PRCS functions as a signal processing unit that processes the signal output from the photoelectric conversion device 10, and may be, for example, a semiconductor device such as a CPU or an ASIC. The display device DSPL may be an EL display device or a liquid crystal display device that displays image data obtained by the photoelectric conversion device 10. The memory device MMRY is a magnetic device or a semiconductor device that stores image data obtained by the photoelectric conversion device 10. The memory device MMRY may be a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive. The mechanical device MCHN has a moving part or a propulsion part such as a motor or an engine. The mechanical device MCHN in the camera can drive parts of the optical system OPT for zooming, focusing, and shutter operation. In the device EQP, the image data output from the photoelectric conversion device 10 is displayed on the display device DSPL, or transmitted to the outside by a communication device (not shown) included in the device EQP. For this reason, the device EQP may include a memory device MMRY and a processing device PRCS.

[0056] A camera incorporating the photoelectric conversion device 10 can be used as a surveillance camera, or an on-board camera mounted on transportation equipment such as automobiles, railroad cars, ships, aircraft, or industrial robots. In addition, a camera incorporating the photoelectric conversion device 10 can be used not only in transportation equipment but also in a wide range of equipment that uses object recognition, such as an intelligent transport system (ITS).

[0057] Next, a specific application in which the photoelectric conversion device 10 (photoelectric conversion system SYS) of the present embodiment is mounted on a moving body will be described with reference to Figs. 11(a) and 11(b). Fig. 11(a) shows an example of a photoelectric conversion system related to an on-board camera. The photoelectric conversion system 8 has a photoelectric conversion device 80. The photoelectric conversion device 80 is the photoelectric conversion device 10 (photoelectric conversion system SYS) described in the above-mentioned embodiment. The photoelectric conversion system 8 has an image processing unit 801 that performs image processing on a plurality of image data acquired by the photoelectric conversion device 80, and a parallax acquisition unit 802 that calculates parallax (phase difference of parallax images) from the plurality of image data acquired by the photoelectric conversion system 8. The photoelectric conversion system 8 also has a distance acquisition unit 803 that calculates a distance to an object based on the calculated parallax, and a collision determination unit 804 that determines whether or not there is a possibility of collision based on the calculated distance. Here, the parallax acquisition unit 802 and the distance acquisition unit 803 are examples of distance information acquisition means that acquire distance information to an object. That is, the distance information is information related to parallax, defocus amount, distance to an object, etc. The collision determination unit 804 may determine the possibility of a collision using any of these pieces of distance information. The distance information acquisition means may be realized by dedicated hardware or a software module. In addition, it may be realized by a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like, or a combination of these.

[0058] The photoelectric conversion system 8 is connected to a vehicle information acquisition device 810, and can acquire vehicle information such as vehicle speed, yaw rate, and steering angle. The photoelectric conversion system 8 is also connected to a control ECU 820, which is a control device that outputs a control signal to generate a braking force for the vehicle based on the judgment result of the collision judgment unit 804. The photoelectric conversion system 8 is also connected to an alarm device 830 that issues an alarm to the driver based on the judgment result of the collision judgment unit 804. For example, when the judgment result of the collision judgment unit 804 indicates that there is a high possibility of a collision, the control ECU 820 performs vehicle control to avoid a collision and reduce damage by applying the brakes, releasing the accelerator, suppressing engine output, etc. The alarm device 830 warns the user by sounding an alarm, displaying alarm information on the screen of a car navigation system, etc., and vibrating the seat belt or steering wheel.

[0059] In this embodiment, the surroundings of the vehicle, for example the front or rear, are imaged by the photoelectric conversion system 8. Fig. 11(b) shows the photoelectric conversion system when imaging the area in front of the vehicle (imaging range 850). A vehicle information acquisition device 810 sends instructions to the photoelectric conversion system 8 or the photoelectric conversion device 80. This configuration can further improve the accuracy of distance measurement.

[0060] The above describes an example of control to avoid collision with other vehicles. However, this is not limited to this, and the invention can also be applied to, for example, control to automatically drive a vehicle by following other vehicles, or control to automatically drive a vehicle without straying from its lane.

[0061] The disclosure of this specification includes the following photoelectric conversion device and photoelectric conversion system.

[0062] (Item 1) A photoelectric conversion device including a plurality of pixels each including a photoelectric conversion element and a plurality of calculation units, the plurality of pixels and the plurality of calculation units are respectively arranged in a two-dimensional array; the plurality of pixels are connected to a corresponding one of the plurality of calculation units for each pixel group formed by two or more pixels among the plurality of pixels; Each of the plurality of pixels outputs a spike-shaped signal to a connected calculation unit among the plurality of calculation units, The photoelectric conversion device, wherein each of the plurality of calculation units performs a calculation on the spike-like signal.

[0063] (Item 2) 2. The photoelectric conversion device according to item 1, further comprising a processing unit that processes calculation results input from each of the plurality of calculation units.

[0064] (Item 3) 3. The photoelectric conversion device according to item 1 or 2, wherein each of the plurality of calculation units includes a neural network.

[0065] (Item 4) 4. The photoelectric conversion device according to item 3, wherein each of the plurality of calculation units performs a calculation based on a membrane potential corresponding to the received spike-like signal.

[0066] (Item 5) 5. The photoelectric conversion device according to any one of items 1 to 4, wherein each of the plurality of pixels outputs the spike-shaped signal in response to a change in the signal output from the photoelectric conversion element.

[0067] (Item 6) 6. The photoelectric conversion device according to item 5, wherein the detection circuit includes a subtraction circuit for obtaining a subtraction value by subtracting a reference value from the signal value of the signal, and a comparison circuit for comparing the subtraction value with a predetermined threshold value.

[0068] (Item 7) 5. The photoelectric conversion device according to any one of items 1 to 4, wherein each of the plurality of pixels outputs the spike-like signal at a frequency corresponding to a signal value of the signal output from the photoelectric conversion element.

[0069] (Item 8) 5. The photoelectric conversion device according to any one of items 1 to 4, wherein the photoelectric conversion element includes an avalanche photodiode.

[0070] (Item 9) The plurality of pixels arranged in a two-dimensional array are thinned out at predetermined intervals, 9. The photoelectric conversion device according to any one of items 1 to 8, wherein each of the plurality of calculation units is disposed in an area where pixels are thinned out.

[0071] (Item 10) The photoelectric conversion device according to any one of items 1 to 8, characterized in that a first semiconductor layer in which at least the photoelectric conversion elements of the plurality of pixels are arranged and a second semiconductor layer in which the plurality of calculation units are arranged are stacked.

[0072] (Item 11) an insulating film is provided on each of bonding surfaces between a first substrate including the first semiconductor layer and a second substrate including the second semiconductor layer, and a connection portion is provided inside the insulating film and made of a metal; The photoelectric conversion device described in item 10, characterized in that the first substrate and the second substrate are bonded to each other by the insulating films of the first substrate and the second substrate, and by the connection portions of the first substrate and the second substrate.

[0073] (Item 12) The plurality of pixels arranged in a two-dimensional array are thinned out at predetermined intervals, 12. The photoelectric conversion device according to item 10 or 11, wherein each of the plurality of calculation units is arranged so as to overlap an area where pixels are thinned out.

[0074] (Item 13) the photoelectric conversion elements arranged in each of the plurality of pixels include a first photoelectric conversion element and a second photoelectric conversion element larger than the first photoelectric conversion element, the plurality of pixels are arranged such that a predetermined number of pixels each including the first photoelectric conversion element are arranged between two pixels each including the second photoelectric conversion element, 12. The photoelectric conversion device according to item 10 or 11, wherein each of the plurality of calculation units is arranged so as to overlap the second photoelectric conversion element among the photoelectric conversion elements arranged in the plurality of pixels.

[0075] (Item 14) A photoelectric conversion device according to any one of items 1 to 13, A computing device; A photoelectric conversion system comprising: The photoelectric conversion system is characterized in that the arithmetic device performs recognition processing based on a signal output from the photoelectric conversion device.

[0076] (Item 15) A photoelectric conversion device according to any one of items 1 to 13, A processing device that processes a signal output from the photoelectric conversion device; An apparatus comprising:

[0077] (Item 16) A moving object including the photoelectric conversion device according to any one of items 1 to 13, A moving body comprising a control device that controls the movement of the moving body using a signal output from the photoelectric conversion device.

[0078] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0079] 10: photoelectric conversion device, C: calculation unit, P: pixel

Claims

1. A photoelectric conversion device comprising a plurality of pixels each including a photoelectric conversion element including an avalanche photodiode, and a plurality of calculation units, the plurality of pixels and the plurality of calculation units are respectively arranged in a two-dimensional array; the plurality of pixels are connected to a corresponding one of the plurality of calculation units for each pixel group formed by two or more pixels among the plurality of pixels, each of the plurality of pixels outputs a spike-like signal to a connected calculation unit among the plurality of calculation units; each of the plurality of calculation units performs a calculation on the spike-like signal; The photoelectric conversion device further comprises a processing unit that processes calculation results input from each of the plurality of calculation units.

2. 2. The photoelectric conversion device according to claim 1, wherein each of the plurality of calculation units includes a neural network.

3. 3. The photoelectric conversion device according to claim 2, wherein each of the plurality of calculation units performs calculations based on membrane potentials corresponding to the received spike-like signals.

4. 2. The photoelectric conversion device according to claim 1, wherein each of the plurality of pixels outputs the spike-like signal in response to a change in the signal output from the photoelectric conversion element.

5. The photoelectric conversion device according to claim 4, characterized in that each of the plurality of pixels includes a subtraction circuit for obtaining a subtraction value obtained by subtracting a reference value from the signal value of the signal output from the photoelectric conversion element, and a comparison circuit for comparing the subtraction value with a predetermined threshold value.

6. 2. The photoelectric conversion device according to claim 1, wherein each of the plurality of pixels outputs the spike-like signal at a frequency corresponding to a signal value of the signal output from the photoelectric conversion element.

7. The plurality of pixels arranged in a two-dimensional array are thinned out at predetermined intervals, 2. The photoelectric conversion device according to claim 1, wherein each of the plurality of calculation units is arranged in an area where pixels are thinned out.

8. The photoelectric conversion device according to claim 1, characterized in that a first semiconductor layer in which at least the photoelectric conversion elements of the plurality of pixels are arranged and a second semiconductor layer in which the plurality of calculation units are arranged are stacked.

9. an insulating film and a connection portion formed of a metal and provided inside the insulating film are provided on each bonding surface between the first substrate including the first semiconductor layer and the second substrate including the second semiconductor layer; The photoelectric conversion device described in claim 8, characterized in that the bonding between the first substrate and the second substrate is achieved by bonding the insulating films of the first substrate and the second substrate together, and by bonding the connecting portions of the first substrate and the second substrate together.

10. The plurality of pixels arranged in a two-dimensional array are thinned out at predetermined intervals, 9. The photoelectric conversion device according to claim 8, wherein each of the plurality of calculation units is arranged so as to overlap an area where pixels are thinned out.

11. the photoelectric conversion elements arranged in each of the plurality of pixels include a first photoelectric conversion element and a second photoelectric conversion element that is larger than the first photoelectric conversion element, the plurality of pixels are arranged such that a predetermined number of pixels each including the first photoelectric conversion element are arranged between two pixels each including the second photoelectric conversion element, 9. The photoelectric conversion device according to claim 8, wherein each of the plurality of calculation units is arranged so as to overlap the second photoelectric conversion element among the photoelectric conversion elements arranged in the plurality of pixels.

12. A photoelectric conversion device comprising a plurality of pixels each including a photoelectric conversion element and a plurality of calculation units, the plurality of pixels and the plurality of calculation units are respectively arranged in a two-dimensional array; the plurality of pixels are connected to a corresponding one of the plurality of calculation units for each pixel group formed by two or more pixels among the plurality of pixels, each of the plurality of pixels outputs a spike-shaped signal to a connected calculation unit among the plurality of calculation units in response to a change in the signal output from the photoelectric conversion element; each of the plurality of calculation units performs a calculation on the spike-like signal; A photoelectric conversion device characterized in that each of the plurality of pixels includes a subtraction circuit for obtaining a subtraction value by subtracting a reference value from the signal value of the signal output from the photoelectric conversion element, and a comparison circuit for comparing the subtraction value with a predetermined threshold value.

13. A first semiconductor layer in which at least the photoelectric conversion elements of the plurality of pixels are arranged and a second semiconductor layer in which the plurality of calculation units are arranged are stacked, The plurality of pixels arranged in a two-dimensional array are thinned out at predetermined intervals, 13. The photoelectric conversion device according to claim 12, wherein each of the plurality of calculation units is arranged so as to overlap an area where pixels are thinned out.

14. The photoelectric conversion elements arranged in each of the plurality of pixels include a first photoelectric conversion element and a second photoelectric conversion element larger than the first photoelectric conversion element; the plurality of pixels are arranged such that a predetermined number of pixels each including the first photoelectric conversion element are arranged between two pixels each including the second photoelectric conversion element, 14. The photoelectric conversion device according to claim 13, wherein each of the plurality of calculation units is arranged so as to overlap the second photoelectric conversion element among the photoelectric conversion elements arranged in the plurality of pixels.

15. A photoelectric conversion device comprising a plurality of pixels each including a photoelectric conversion element and a plurality of calculation units, the plurality of pixels and the plurality of calculation units are respectively arranged in a two-dimensional array; the plurality of pixels are connected to a corresponding one of the plurality of calculation units for each pixel group formed by two or more pixels among the plurality of pixels, each of the plurality of pixels outputs a spike-like signal to a connected calculation unit among the plurality of calculation units; each of the plurality of calculation units performs a calculation on the spike-like signal; The plurality of pixels arranged in a two-dimensional array are thinned out at predetermined intervals, A photoelectric conversion device, characterized in that each of the plurality of calculation units is arranged in an area where pixels are thinned out.

16. A photoelectric conversion device comprising a plurality of pixels each including a photoelectric conversion element and a plurality of calculation units, the plurality of pixels and the plurality of calculation units are respectively arranged in a two-dimensional array; the plurality of pixels are connected to a corresponding one of the plurality of calculation units for each pixel group formed by two or more pixels among the plurality of pixels, each of the plurality of pixels outputs a spike-like signal to a connected calculation unit among the plurality of calculation units; each of the plurality of calculation units performs a calculation on the spike-like signal; a first semiconductor layer in which at least the photoelectric conversion elements of the plurality of pixels are arranged, and a second semiconductor layer in which the plurality of calculation units are arranged are stacked; the photoelectric conversion elements arranged in each of the plurality of pixels include a first photoelectric conversion element and a second photoelectric conversion element that is larger than the first photoelectric conversion element, the plurality of pixels are arranged such that a predetermined number of pixels each including the first photoelectric conversion element are arranged between two pixels each including the second photoelectric conversion element, a second photoelectric conversion element disposed in each of the plurality of pixels, the second photoelectric conversion element being overlapped with the first photoelectric conversion element;

17. The photoelectric conversion device according to any one of claims 1 to 16, A computing device; A photoelectric conversion system comprising: The photoelectric conversion system is characterized in that the arithmetic device performs recognition processing based on the signal output from the photoelectric conversion device.

18. The photoelectric conversion device according to any one of claims 1 to 16, a processing device that processes a signal output from the photoelectric conversion device; An apparatus characterized by comprising:

19. A moving object comprising the photoelectric conversion device according to any one of claims 1 to 16, A moving body comprising a control device that controls the movement of the moving body using a signal output from the photoelectric conversion device.

Images