Photoelectric conversion device, photoelectric conversion system

Abstract

To provide a photoelectric conversion device that speeds up the generation of control signals and can obtain signal processing results that are unlikely to cause a reduction in the signal-to-noise ratio.SOLUTION: A photoelectric conversion device includes a first substrate having pixels including photoelectric conversion elements and a second substrate having a first control unit including a first signal processing unit for processing the signals from the pixels and stacked on the first substrate. The signals from the pixels are output to a second signal processing unit arranged in a location different from the first signal processing unit. A pathway in which signals from the pixels are output to the first signal processing unit is different from a pathway in which signals from the pixels are output to the second signal processing unit. The first control unit controls the pixels on the basis of the signals processed by the first signal processing unit.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] It relates to a photoelectric conversion device and a photoelectric conversion system.

Background Art

[0002] In a photoelectric conversion device, higher functionality and the like are required. For example, it is required to enable a higher dynamic range, detection of a high-speed moving object, etc., and to increase the added value of the photoelectric conversion device.

[0003] Patent Document 1 describes an image sensor in which a first substrate and a second substrate are laminated. Patent Document 1 discloses that pixels are arranged on the first substrate, and a readout circuit for processing pixel signals of the first substrate and a control circuit are arranged on the second substrate. Further, Patent Document 1 discloses that the control circuit receives the processing result in the readout circuit, and the control circuit controls the pixels for each pixel group based on the processing result in the readout circuit.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Control signals generated based on the processing results of pixel signals may need to be generated quickly depending on the application. For example, in the case of subjects with changing brightness or subjects moving at a speed exceeding a predetermined speed, the generated control signals may be inappropriate due to delays caused by the generation time of the control signals. Therefore, it is necessary to speed up the generation of control signals. On the other hand, in order to obtain signal processing results that do not cause a decrease in the signal-to-noise ratio, amplification of the pixel signals and high-resolution AD conversion signal processing are necessary. Therefore, there is a trade-off between speeding up the generation of control signals and obtaining signal processing results that do not cause a decrease in the signal-to-noise ratio.

[0006] The objective is to provide a photoelectric converter that can generate control signals at high speed and obtain signal processing results that do not cause a decrease in the signal-to-noise ratio. [Means for solving the problem]

[0007] The present invention has been made in view of the above problems, and one embodiment comprises a first substrate having pixels including photoelectric conversion elements, and a second substrate laminated with the first substrate having a first control unit including a first signal processing unit that processes signals from the pixels, wherein the signals from the pixels are output to a second signal processing unit located at a different position from the first signal processing unit, the path through which the signals from the pixels are output to the first signal processing unit is different from the path through which the signals from the pixels are output to the second signal processing unit, and the first control unit controls the pixels based on the signals processed by the first signal processing unit. [Effects of the Invention]

[0008] This provides a photoelectric converter that can generate control signals at high speed and obtain signal processing results that are less prone to a decrease in the signal-to-noise ratio. [Brief explanation of the drawing]

[0009] [Figure 1] Schematic perspective view of the photoelectric converter of the first embodiment. [Figure 2] Plan view schematic diagram of the photoelectric converter of the first embodiment [Figure 3] Plan view schematic of a photoelectric converter according to a modified example of the first embodiment [Figure 4] Circuit diagram of the photoelectric converter of the first embodiment [Figure 5] Circuit diagram of the photoelectric converter of the first embodiment [Figure 6] Drive timing of the photoelectric converter in the first embodiment [Figure 7] Functional block diagram illustrating the outline of the first embodiment. [Figure 8] Circuit diagram of the photoelectric converter of the second embodiment [Figure 9] A schematic plan view showing an example of a photoelectric conversion element for a pixel in the second embodiment. [Figure 10] Circuit diagram of the photoelectric converter of the third embodiment [Figure 11] Functional block diagram of the photoelectric conversion system of the fourth embodiment [Figure 12] Functional block diagram of the photoelectric conversion system of the fifth embodiment [Figure 13] Functional block diagram of the photoelectric conversion system of the sixth embodiment [Figure 14] Diagram of the photoelectric conversion system and mobile body according to the seventh embodiment. [Figure 15] Schematic diagram of the photoelectric conversion system of the eighth embodiment [Figure 16] Functional block diagram of the photoelectric conversion system of the ninth embodiment [Modes for carrying out the invention]

[0010] The embodiments will be described below with reference to the drawings. The embodiments shown below are intended to embody the technical concept of the present invention and do not limit the present invention. The size and positional relationships of the components shown in each drawing may be exaggerated for clarity of explanation. In the following description, identical components may be given the same number and their explanation may be omitted.

[0011] In each of the embodiments described below, as an example of a photoelectric conversion device, an imaging device will be mainly described. However, each embodiment is not limited to an imaging device. The photoelectric conversion device may be, in addition to an imaging device, a distance measuring device (a device for distance measurement using focus detection or TOF (Time Of Flight)), a photometric device (a device for measuring the amount of incident light), or the like.

[0012] In this specification, "planar view" refers to viewing from a direction perpendicular to the light incident surface of the first substrate described later. When the light incident surface of the first substrate is rough when viewed microscopically, the planer view is defined based on the light incident surface of the first substrate when viewed macroscopically.

[0013] (First Embodiment) The first embodiment of the present disclosure will be described with reference to the drawings.

[0014] FIG. 1 is a diagram showing a schematic perspective view of the photoelectric conversion device of this embodiment. The photoelectric conversion device of this embodiment is composed of a first substrate 10 and a second substrate 20 laminated and bonded to each other. The bonding of these two substrates is, for example, a metal bonding in which metal members arranged in the insulating layers of the first substrate 10 and the second substrate 20 are joined to each other. The metal bonding portion where the metal members are connected is referred to as a connection portion 400. As another bonding method between the first substrate 10 and the second substrate 20, bonding using bumps may be employed.

[0015] Further, pads 300 are arranged on the second substrate 20, and openings 200 are provided on the first substrate 10 corresponding to the positions of the pads 300. The pads 300 and a circuit outside the photoelectric conversion device are connected via wire bonding. Then, reception of an input signal and transmission of an output signal are performed via the pads 300.

[0016] In FIG. 1, for easy understanding, the connection portion 400 is shown on the first surface of the first substrate 10, but the connection portion 400 is arranged on the second surface of the first substrate 10 and cannot be visually recognized from the first surface side of the first substrate 10. Also, in FIG. 1, six connection portions 400 are shown, but the number and arrangement positions of the connection portions 400 are not limited to this.

[0017] Furthermore, the pad 300 does not need to be located on the second board 20; it may be located on the first board 10.

[0018] First, the basic configuration of this embodiment will be described with reference to Figures 2(A), 2(B), 4, 5, and 6, and the drive mechanism will be described with reference to Figure 7.

[0019] Figures 2(A) and 2(B) are schematic plan views of the photoelectric converter of this embodiment. In Figures 2(A) and 2(B), the opening 200 and pad 300, which were described in Figure 1, are omitted for the sake of simplifying the drawings.

[0020] Figure 2(A) shows a schematic planar layout of the first substrate 10. Figure 2(B) shows a schematic planar layout of the second substrate 20. In Figure 2(A), an example is shown in which the first substrate 10 is arranged with pixels including photoelectric conversion elements and a signal processing unit 500 that mainly processes signals output from the pixels (hereinafter referred to as pixel signals). The second substrate 20 has a pixel signal processing control unit 711 which includes a pixel signal processing unit that processes pixel signals, a drive timing control unit 700 which controls the driving of the circuits arranged on the first substrate 10, and digital signal processing circuits. These circuits are arranged in a different location from the signal processing unit 500 and have different functions from the signal processing unit 500.

[0021] The following description will mainly focus on the arrangement of the components shown in Figures 2(A) and 2(B), but the arrangement of these components is not limited to the configurations shown in Figures 2(A) and 2(B). For example, as shown in Figure 3(A), a pixel section 100 containing multiple pixels may be arranged on the first substrate 10, and as shown in Figure 3(B), peripheral circuits such as a signal processing unit 500 and a vertical scanning circuit may be arranged on the second substrate 20. Note that Figure 3(B) is merely an example of the arrangement of the AD conversion circuit, and the arrangement of the signal processing circuit 500, etc., is not limited to the arrangement shown in Figure 3(B). Furthermore, a timing generation circuit and part of the digital signal processing circuit may be provided on the first substrate 10.

[0022] As mentioned above, the first substrate 10 and the second substrate 20 are connected by a connecting portion 400. In the planar layout, the connecting portion 400 is positioned at the same coordinates on both the first substrate 10 and the second substrate 20. That is, in the orthogonal projection onto the first surface of the first substrate 10, the first substrate 10, the connecting portion 400, and the second substrate 20 are positioned in overlapping locations. In the following description, the connecting portions 400 positioned on the first substrate 10 and the second substrate 20 will not be distinguished. When referred to as connecting portion 400-1, it refers to connecting portion 400-1 positioned on either the first substrate 10 or the second substrate 20. Also, components with the same function will be listed without sub-numbers. For example, when referred to as connecting portion 400, connecting portion 400-n (where n is a natural number) will be listed without distinction.

[0023] In the drawings, wiring and connection points 400 are shown as single elements. In reality, multiple wirings and multiple connection points 400 may be arranged, such as multiple power lines, multiple grounding lines, multiple control lines, and multiple signal input / output lines. Furthermore, even when the same power or signal is supplied, multiple connection points 400 may be arranged to reduce resistance and provide redundancy. In the following explanation, even if multiple wirings or connection points should be arranged, they may be omitted in order to simplify the drawings and explanations.

[0024] In Figures 2(A) and 2(B), the components and wiring are sometimes shown in a configuration where they do not physically overlap in a plan view; however, in reality, the components may overlap. Similarly, the placement of the connection section 400 can be freely designed in relation to the components and wiring.

[0025] Drive signals that control the circuits that process pixels and pixel signals, and processed pixel signals are transmitted between the first substrate 10 and the second substrate 20 via multiple connection points 400. First, the components arranged on the first substrate 10 in Figure 2(A) will be described, and then, in the configuration with the substrates bonded together, the components arranged on the second substrate 20 in Figure 2(B) will be described, as well as the signal transmission, control, and driving between the substrates.

[0026] The configuration of the first substrate 10 will be explained using Figures 2(A), 4, and 5. The pixel section 100 has pixels 101, including the photoelectric conversion element 102 shown in Figure 3, arranged in two dimensions. Each pixel 101 converts an optical signal into an electrical signal and outputs it from the pixel section 100. In this embodiment, the configuration allows for drive control in block units containing multiple pixels 101. Multiple pixels 101 that are driven and controlled in block units are collectively called a pixel block. The pixel section 100 has, for example, 8 x 8 pixel blocks totaling 64 pixels, as shown by the dotted lines. In Figure 2(A), only one pixel 101 is shown in one pixel block, but a pixel block is assumed to contain multiple pixels 101. The number of pixel blocks, the way the pixel blocks are divided, the number of pixels 101 included in a pixel block, etc., can be changed as appropriate.

[0027] By configuring the pixels 101, which are arranged in a two-dimensional shape in this way, to be controlled in block units, optimal control can be performed for each block according to the brightness of the subject.

[0028] The pixel 101 shown in Figure 4 has a power supply potential VDD, a ground potential GND, a photoelectric conversion element 102, a transfer transistor 103, a reset transistor 104, an amplification transistor 105, a selection transistor 106, and an output line 109. It also has a capacitive selection transistor 107 and a capacitor 108, which can switch the charge-voltage conversion gain of the pixel. The transfer transistor 103 is controlled on / off by the control signal PTX. The reset transistor 104 is controlled on / off by the control signal PRES. The selection transistor 106 is controlled on / off by the control signal PSEL. The capacitive selection transistor 107 is controlled on / off by the control signal FDG. The output line 109 is connected to a circuit located outside the pixel section 100, to which multiple pixels 101 of the corresponding column are connected. The power supply potential VDD and ground potential GND are supplied from power lines (not shown). The power supply potential VDD and ground potential GND are supplied to the power lines via the pad 300 shown in Figure 1. The accumulation operation of pixel 101 begins by simultaneously turning on the transfer transistor 103 and the reset transistor 104 to reset the photoelectric conversion element 102, and then turning off the transfer transistor 103. The timing of turning off the transfer transistor 103 marks the start of accumulation. Subsequently, the transfer transistor 103 is turned on, and the accumulation time continues until the charge accumulated in the photoelectric conversion element 102 is transferred to the floating diffusion region (FD) connected to the gate electrode of the amplification transistor 105. Furthermore, the effective capacitance value of the floating diffusion region can be changed by turning the capacitance selection transistor 107 on and off. The charge transferred to the floating diffusion region is output to the circuit outside the pixel section 100 via the amplification transistor 105, the selection transistor 106, and the output line 109.

[0029] The pixel control unit 120 controls the pixels 101 via the pixel control lines 121. The pixel control unit 120 is, for example, a vertical scanning circuit. In Figure 2(A), the pixel control line 121 is shown as a single line for simplification of the drawing, but it includes multiple control lines, including control signals PTX, PRES, PSEL, and FDG. Although the reference numerals are omitted, the configuration is such that each of the 8x8 pixel blocks has a pixel control line. However, the control signals shown in the drawing are not particularly distinguished as control for each pixel, each row of pixels, or each pixel block.

[0030] The pixel signal output from the pixel unit 100 is input to the signal processing unit 500 via the wiring 140 shown as output line 109 in Figure 4. The signal processing unit 500 includes multiple circuits for processing the pixel signal. The signal processing unit 500 is also configured to be controllable in block units, including multiple processing circuits corresponding to the pixel block unit, as shown by the dotted line.

[0031] The signal processing unit 500 includes, for example, a constant current circuit. The amplification transistor 105 of the pixel 101 forms a source follower circuit by connecting the constant current circuit via the selection transistor 106, output line 109, and wiring 140. The formation of this source follower circuit allows the pixel signal to be read from the pixel unit 100. The output of the source follower circuit may be further amplified by an amplifier. Figure 5 is an equivalent circuit diagram of an amplifier 501 that amplifies the pixel signal. The amplifier 501 consists of an amplification circuit 510, an input capacitor C0, feedback capacitors C1 and C2, and switches M0 and M1. Switch M0 is controlled by a reset control line 503, and switch M1 is controlled by a gain control line 502. The voltage amplification factor of the amplifier 501 is expressed as -input capacitor C0 / feedback capacitors (C1, C2). In a system where negative feedback is applied to the amplification circuit 510, the voltage amplification factor, determined by the voltage division ratio between the feedback capacitors C1 and C2 and the input capacitor C0, can be switched by selecting either feedback capacitor C1 or C1+C2 using switch M1. The negative sign in the voltage amplification factor indicates that it is an inverting amplifier circuit. Switch M0 is a switch that shorts the input and output of the amplification circuit 510, and by turning it on before reading the pixel signal and performing amplification operation, the amplifier 501 is reset. After the reset is released, the pixel signal output from the source follower circuit is amplified by a voltage amplification factor of -C0 / C1 when switch M1 is off, and output from the amplified signal output line 504. When switch M1 is on, it is amplified by a voltage amplification factor of -C0 / (C1+C2). In this embodiment, there is one type of input capacitor C0 and two types of feedback capacitors C1 and C2, but it is also possible to provide multiple input capacitors and feedback capacitors to set the voltage amplification factor.

[0032] The signal processing unit 500 may further include multiple signal processing functions. In this embodiment, the signal processing unit 500 has an analog-to-digital conversion circuit (AD conversion circuit). The pixel signal amplified by the amplifier 501 is converted into a digital signal in the AD conversion circuit. The AD conversion result is stored in the memory unit 520 via the amplified signal output line 504. The signal processing unit 500 may also include a circuit having a CDS (Correlated Double Sampling) function and a sample-and-hold (S / H) function for the pixel signal. Furthermore, the signal processing unit 500 may include a circuit that generates a reference power supply and a clock signal used in the circuits constituting the signal processing unit 500. The signal processing unit 500 is supplied with a power supply potential VDD and a ground potential GND from a power supply line not shown. Note that the ground potential GND of the pixel unit and the ground potential GND of the signal processing unit 500 can be supplied by separate power supplies.

[0033] The signal processing unit 500 is controlled by control lines 505 and 506. Control lines 505 and 506 are a plurality of control lines, including the aforementioned gain control line 502 and reset control line 503. In this embodiment, control line 505 is a common control line for multiple signal processing circuits included in the signal processing unit 500. On the other hand, control line 506 is a control line for each block of multiple block-based signal processing circuits. For example, control line 506 includes a gain control line 502, making it possible to control the gain of the amplifier 501 for each block. Control lines 505 and 506 are generated by a control signal generation unit (not shown) based on control signals input from the drive timing control unit 700 located on the second board 20 (described later) via control lines 704 and 714 and connection parts 400-3 and 400-7. The control signal generation unit includes a voltage level shift circuit, a buffer circuit, a decoder, etc.

[0034] The memory unit 520 includes a digital memory for holding the AD conversion results and a scanning circuit for reading the digital data held in the digital memory. The memory unit 520 is controlled based on control signals input from the drive timing control unit 700 via control line 706, connection unit 400-4, and wiring 521. The digital data held in the memory unit 520 is output to the second board 20 via wiring 522 and connection unit 400-5.

[0035] Next, the configuration of the second circuit board 20 in Figure 2(B) will be described. Furthermore, the signal transmission and control relationships with the first circuit board 10 via the connection section 400 will be explained.

[0036] The drive timing control unit 700 is a circuit that controls the entire photoelectric conversion device. The control line 701 is a wire that transmits control signals related to the control of the pixel unit 100, and is connected to the pixel control unit 120 via the connection part 400-1 and the wire 123. Similarly, the control line 702 is also a wire that transmits control signals related to the control of the pixel unit 100, and is connected to the pixel unit 100 via the connection part 400-2 and the pixel control line 122.

[0037] The control unit 710 includes a circuit for controlling the pixel section 100 and the signal processing unit 500 of the first substrate 10. In this embodiment, multiple pixels 101 and the signal processing unit 500 are controlled in pixel block units. One example of control in pixel block units is the control of the pixel storage time. In this embodiment, a connection section 400-6 is provided at a position overlapping the pixel section 100 in the orthogonal projection onto the first surface of the first substrate 10, and the storage time of each pixel block can be controlled via the connection section 400-6.

[0038] The pixel storage time is controlled by a pixel signal processing control unit 711, which is arranged in units of pixel blocks. The pixel signal processing control unit 711 generates a storage control signal based on a signal supplied from the drive timing control unit 700 via the control line 703. The storage control signal generated by the pixel signal processing control unit 711 is connected to the pixel 101 via wiring 712, connection part 400-6, and wiring 130.

[0039] Note that in Figures 2(A) and (B), the pixel signal processing control unit 711 and connection unit 400-6 are shown in only one block for the sake of simplifying the diagram, but they are actually located in all blocks. Furthermore, the storage time control of the pixel unit 100 may be controlled not only by the pixel signal processing control unit 711, but also by the drive timing control unit 700 and the pixel control unit 120. Similarly, other controls related to pixels may be driven by any of the drive timing control unit 700, the pixel control unit 120, or the pixel signal processing control unit 711.

[0040] The signal processing unit 500 is controlled block by block from the signal processing control unit 713, wiring 714, connection unit 400-7, and control line 506. As explained earlier, one of the functions of the signal processing unit 500 that is controlled block by block is the control of the voltage amplification factor of the amplifier 501. Other controls include the drive current control of the amplification circuit 510, power-on control, and power-down control. In addition, the configuration may include controlling the AD conversion operation and the analog-to-digital conversion gain (AD conversion gain) as functions controlled block by block.

[0041] The pixel signals, which have been converted from digital to digital by the signal processing unit 500 and stored in the memory unit 520, are input to the digital signal processing unit 720 via wiring 522, connection unit 400-5, and wiring 722. The digital signal processing unit 720 performs various digital signal processing on the input digital signals, such as noise reduction processing, digital gain processing, offset addition / subtraction, compression processing, and data scrambling processing. The digital data processed by the digital signal processing unit 720, or a portion thereof, is input to the digital signal processing control unit 730 via wiring 721.

[0042] The digital signal processing control unit 730 has the function of calculating the optimal storage time for each pixel block, the voltage amplification factor for each block of the signal processing unit 500, etc., based on the pixel signals processed by the signal processing unit 500. Note that the storage time may be controlled indirectly by controlling the control unit 710, rather than directly controlling the pixels from the digital signal processing control unit 730. Alternatively, the storage time may be controlled by both the control unit 700 and the digital signal processing control unit 730. The control unit 750 may include some functions of the drive timing control unit 700. Furthermore, the drive timing control unit 700 may include the control unit 750. Control information regarding the settings of the storage time and voltage amplification factor calculated by the digital signal processing control unit 730 is input to the control unit 750 via wiring 723. The control unit 750 generates a storage control signal for pixel 101 and a voltage amplification factor control signal for amplifier 501, and these generated control signals are input to the control unit 710 via wiring 751. In the control unit 710, the pixel signal processing control unit 711 controls the pixel 101, and the signal processing control unit 713 controls the signal processing unit 500. In addition, the control information for storage time and voltage amplification ratio calculated by the digital signal processing control unit 730 is also input to the digital signal processing unit 720 via wiring 723. Based on the control information calculated by the digital signal processing control unit 730, the digital signal processing unit 720 can process the pixel signal input from the memory unit 520 with appropriate parameters.

[0043] The digital signal processing control unit 730 is connected to the memory unit 740 via wiring 731. In the process of calculating the storage time and voltage amplification factor in the digital signal processing control unit 730, a program stored in the memory unit 740 is executed to perform various processes using a trained model created by machine learning using a deep neural network. This trained model (neural network computation model) may be designed based on parameters generated by inputting input signals corresponding to the output of the pixel unit 100 and training data, which are linked to labels for said input signals, into a predetermined machine learning model. The predetermined machine learning model may also be a learning model using a multi-layer neural network (also called a multi-layer neural network model). The calculation process is, for example, a process of multiplying the dictionary coefficients of the trained model by the image data obtained from the pixel unit 100. The results obtained by such calculation processing (calculation results) can be output to the output interface unit. The calculation results may include image data obtained by performing calculation processing using the trained model, and various information (metadata) obtained from that image data. Furthermore, the control unit 710 is connected to the memory unit 740 by wiring 715, and, like the digital signal processing control unit 730, is configured to perform various signal processing operations.

[0044] The pixel signals processed digitally by the digital signal processing unit 720 are output to the outside of the photoelectric converter via the output unit 800 through the wiring 724. The output method of the output unit 800 can be, for example, a method that outputs voltage from a single terminal, such as a buffer circuit, or an LVDS (Low Voltage Differential Signaling) method with two differential terminals. Furthermore, if the digital signals input from the digital signal processing unit 720 are parallel data, it may have, for example, a parallel-to-serial conversion (P / S conversion) function.

[0045] The pixel signal processing control unit 711, which processes pixel signals, has the function of processing pixel signals in addition to the function of controlling the pixel storage time based on control signals input from the drive timing control unit 700 and the control unit 750. For example, the pixel signal processing control unit 711 has an AD conversion circuit with a lower resolution than the AD conversion circuit included in the signal processing unit 500, and can process pixel signals input via the connection unit 400-6. The pixel signal processing control unit 711 is configured to be able to determine the approximate brightness of the subject. Furthermore, it is configured to have a function that can control the pixels 101 based on the brightness information of the subject.

[0046] As a specific example of the configuration, for instance, the low-resolution AD conversion circuit is a 1-bit AD conversion circuit that determines the magnitude of the signal level relative to a specific reference voltage, and is configured to determine the magnitude of the brightness of the subject relative to a specific brightness. Furthermore, the configuration allows for the control of the capacitance selection transistor 107 of the pixel 101 based on the result of the subject brightness determination.

[0047] For example, if the brightness of the subject is above a certain level and the capacitance selection transistor 107 of pixel 101 is off, the amplitude of the output line 109 may become large and exceed the input range of the signal processing unit 500. As a result, the image becomes low resolution and the subject cannot be identified.

[0048] In contrast, in this embodiment, in such cases, based on the 1-bit AD conversion result, the pixel 101 and the capacitance selection transistor 107 are switched from the off state to the on state, and the capacitance 108 is connected to the gate of the amplification transistor 105. This effectively increases the capacitance of the floating diffusion region and reduces the amplitude of the output line 109. As a result, the signal amplitude can be appropriately adjusted with respect to the input range of the signal processing unit 500.

[0049] As shown in the configuration of this embodiment, 1-bit AD conversion or low-resolution AD conversion can be processed at high speed, and the pixel 101 can be controlled before the signal processing of the signal processing unit 500 begins.

[0050] Furthermore, for each pixel block, the pixels monitored by the pixel signal processing control unit 711, the number of pixels, and the resolution of the AD conversion circuit can be changed as appropriate. Additionally, the signal processing control unit 713 may control the voltage amplification factor of the amplifier 501 and the AD conversion gain of the AD conversion circuit based on the brightness determination result of the pixel signal processing control unit 711. The signal processing results of the pixel signal processing control unit 711 can also be input to the digital signal processing unit 720 and the digital signal processing control unit 730 via wiring 751, control unit 750, and wiring 723, allowing them to be reflected in various digital signal processing operations.

[0051] By individually setting and controlling the storage time of pixel 101 and the voltage amplification factor of amplifier 501 for each pixel block from the drive timing control unit 700, appropriate settings can be made according to the brightness of each block, even for subjects with a wide brightness range. This makes it possible to obtain images with a wide dynamic range. In addition, the digital signal processing control unit 730 updates and generates control information in real time based on subject information. For example, even when the brightness of the subject changes continuously, such as in video recording, optimal control can be made for each block in conjunction with the change in subject brightness.

[0052] Figures 6(A) and 6(B) illustrate the drive timing of the photoelectric conversion device in this embodiment. In Figures 6(A) and 6(B), transistors and switches are turned on when the pulse is high level and turned off when the pulse is low level. "TX" is the period during which charge is transferred from the photoelectric conversion element 102 to the floating diffusion region. Charge is transferred from the photoelectric conversion element 102 to the floating diffusion region during the period when the control signal PTX is high level. "AD" is the period during which the signal processing unit 500 amplifies the pixel signal of the amplifier 501 and performs AD conversion. "DSP1" is the period during which the digital signal processing control unit 730 generates various calculations and control signals based on the digital data input from the digital signal processing unit 720. "DSP2" is the period during which the pixel signal processing control unit 711 performs AD conversion of the pixel signal input from the pixel 101 with low resolution and generates a control signal based on the AD conversion result. The voltage amplification factor of the amplifier 501 is represented by Gain1 and Gain2. Gain2 has a higher voltage amplification factor compared to Gain1.

[0053] Figure 6(A) shows a comparative configuration in which the photoelectric converter is driven based solely on the calculations and control of the digital signal processing control unit 730, and the pixel signal processing control unit 711 is not operated. In this case, based on the calculations and control signals of "DSP1" in the N-1 (N: natural number) frame, the settings for the storage time of pixel 101 and the voltage amplification factor of amplifier 501 are reflected in frames from the Nth frame onward. In the driving example in Figure 6(A), if the brightness of a certain pixel block is determined to be low in the N-1 frame, the settings are controlled to either increase the storage time, increase the voltage amplification factor, or deselect capacitor 108 in frames from the Nth frame onward. Therefore, in Figure 6(A), the voltage amplification factor is controlled from Gain1 to Gain2 in the Nth frame, and the capacitance 108 of pixel 101 is controlled from selected to deselected by the control signal FDG.

[0054] Figure 5(B) shows an example of driving the imaging device based on calculations and controls by the digital signal processing control unit 730 and the pixel signal processing control unit 711. In the driving example in Figure 5(B), similar to Figure 5(A), the settings for the storage time of pixel 101 and the voltage amplification factor of amplifier 501 are reflected in the Nth frame based on the calculations and control signals of "DSP1" in the N-1th frame. In the Nth frame, the "TX" operation period is simultaneously set to the "DSP2" operation period. Due to the "DSP2" operation, the capacitance 108 of pixel 101 is controlled from unselected to selected by the control signal FDG based on the pixel signal of the Nth frame, and the voltage amplification factor of amplifier 501 is controlled from Gain2 to Gain1.

[0055] In the case of Figure 6(A), if the brightness of the subject changes to a high brightness in the Nth frame or later, when the settings are applied, the signal read from pixel 101 may exceed the input range of the signal processing unit 500, and a correct image may not be obtained. Conversely, if the brightness is judged to be high in the N-1 frame, the settings may be calculated to shorten the storage time and decrease the voltage amplification factor in the Nth frame and later. If the brightness of the subject changes to a low brightness in the Nth frame or later, when the settings are applied, the signal read from pixel 101 may be small, and the pixel signal may be buried in noise. Similarly, in the case of a subject whose brightness changes rapidly and continuously, the storage time and voltage amplification factor calculated and set by the digital signal processing control unit 730 may not be set appropriately for the subject. These can be reduced by increasing the pixel signal readout operation and the signal processing speed of the signal processing unit 500. On the other hand, obtaining an image with a high signal-to-noise ratio requires a high response time for the pixel signal and high-resolution A / D conversion, which necessitates appropriate signal processing time.

[0056] In this embodiment, the path through which the signal from a pixel is output to the first signal processing unit is different from the path through which the signal from a pixel is output to the second signal processing unit. For example, as shown in Figure 7, the signal from the pixel generated and output by the pixel included in the pixel unit 100 is output to both the signal processing unit 500 (second signal processing unit) and the control unit 710, which includes the pixel signal processing unit (first signal processing unit) of the control unit 710. In other words, there is a path P1 that outputs to the signal processing unit 500 and a path P2 that outputs to the signal processing unit. The pixel is then controlled based on the processing result of the output from path P2 to the signal processing unit. According to this embodiment, the pixel signal output through path P2 is processed by the signal control unit, and the pixel of the pixel unit 100 is controlled based on the processing result. On the other hand, signal processing such as AD conversion can be performed based on the pixel signal output through path P1. Therefore, the generation of control signals can be accelerated, and it is possible to obtain signal processing results that do not cause a decrease in the signal-to-noise ratio.

[0057] Furthermore, according to this embodiment, the pixel signal processing control unit 711 can monitor the pixel signal of its own frame, determine the brightness, and control the pixel 101 and the signal processing unit 500 within the same frame. If the settings set in the digital signal processing control unit 730 are inappropriate, the pixel 101 and the signal processing unit 500 are controlled, so that signal processing is performed within an appropriate range. Therefore, for subjects with a wide brightness range, an image with a wide dynamic range can be obtained regardless of the brightness changes or movement speed of the subject.

[0058] (Second Embodiment) Figure 8 is a circuit diagram of a pixel in a photoelectric converter according to the second embodiment of the present invention, and Figure 9 is a schematic plan view of the two photoelectric converter elements included in the pixel of this embodiment. The photoelectric converter according to this embodiment differs from the first embodiment in that it has two photoelectric converter elements. Here, we will mainly explain the differences from the first embodiment. Except for the matters described below, it is substantially the same as the first embodiment, so the same reference numerals are used for components that are the same as in the first embodiment and their descriptions may be omitted.

[0059] The pixel 101 shown in Figure 8 is equipped with a photoelectric conversion element 110 having the same or different sensitivity as the photoelectric conversion element 102, and is connected to the gate of the amplification transistor 105 via a transfer transistor 111. The transfer transistors 103 and 111 are controlled by control lines PTX1 and PTX2, respectively. PTX1 and PTX2 are controlled according to the operating mode.

[0060] Furthermore, the photoelectric conversion elements 102 and 110 may have the same light-receiving area and light-shielding pattern in their planar layout, or they may have different configurations. Figure 9 shows an example of a photoelectric conversion element 110, in which the sensitivity is changed by varying the area from that of photoelectric conversion element 102. In terms of the process, the formation conditions may be the same, or they may have different configurations.

[0061] Furthermore, the color filters placed on the light-receiving surfaces of the photoelectric conversion elements 102 and 110 may be of the same color or different colors. In addition, the photoelectric conversion elements 102 and 110 may be arranged on a single microlens to enable the detection of phase differences.

[0062] In this embodiment, one operating mode is to set the drive timing of control lines PTX1 and PTX2 to the same timing, thereby reading out the photoelectric conversion elements 102 and 110 simultaneously with the same storage time, and obtaining a highly sensitive image. Another example of an operating mode is to read out at different timings with the same storage time, thereby increasing the resolution of the readout as separate pixels, or enabling phase detection from the difference between the two signals. Furthermore, another example of an operating mode is to read out the photoelectric conversion elements 102 and 110 at different timings with different storage times and combine them in subsequent signal processing, thereby obtaining an image with a wide dynamic range.

[0063] These operating modes may be controlled by the signal processing of the pixel signal processing control unit 711 and the digital signal processing control unit 730. For example, the other photoelectric conversion element is controlled according to the pixel signal level read from one of the photoelectric conversion elements 102 and 110. If the pixel signal level read from one photoelectric conversion element is above a predetermined value, the reading timing of the other photoelectric conversion element is made different, allowing it to be processed as a separate pixel. In addition to processing it as a separate pixel, it can also be controlled to be used for phase difference detection or not read at all. Furthermore, if the pixel signal level read from one photoelectric conversion element is below a predetermined value, a highly sensitive image can be obtained by simultaneously reading the other photoelectric conversion element. As in the first embodiment, the signal processing settings of the signal processing unit 500 and the digital signal processing unit 720 are linked to the operating mode based on the processing results of the pixel signal processing control unit 711 and the digital signal processing control unit 730, thereby obtaining an appropriate signal processing result.

[0064] According to this embodiment, similar to the first embodiment, it is possible to obtain signal processing results that enable faster generation of control signals and are less prone to a decrease in the signal-to-noise ratio. Furthermore, in this embodiment, multiple photoelectric conversion elements are provided in the pixel 101, and the operation mode can be switched for each pixel 100 or pixel block by signal processing and control of the pixel signal processing control unit 711 and the digital signal processing control unit 730. This makes it possible to set optimal imaging conditions for the subject in terms of dynamic range, power consumption, etc., when using the photoelectric conversion device as an imaging device. In addition, a variety of applications can be realized by combining this with control from outside the photoelectric conversion device.

[0065] (Third embodiment) Figure 10 shows a circuit diagram of a pixel in a photoelectric conversion device according to a third embodiment of the present invention. This embodiment differs from the first embodiment in that the anode terminal voltage of the photoelectric conversion element can be switched. This section will focus on explaining the differences from the first embodiment. Except for the matters described below, this embodiment is substantially the same as the first embodiment, so components similar to those in the first embodiment are denoted by the same reference numerals and their descriptions may be omitted.

[0066] In Figure 10, the pixel 101 has a switch S10 placed between the anode of the photoelectric conversion element 102 and a signal line to which a reference voltage VA is supplied, and a switch S11 placed between the anode and a signal line to which a ground potential is supplied. The voltage applied to the anode can be switched between the reference voltage VA and the ground potential GND by controlling the on / off state of switches S10 and S11, respectively. The reference voltage VA is a voltage that can reverse bias the photoelectric conversion element 102, for example, a negative voltage. The reference voltage VA is, for example, -20V. When switch S10 is on and switch S11 is off, a reverse bias voltage that causes avalanche multiplication is supplied to the photoelectric conversion element 102, and when a photon is incident, avalanche breakdown occurs. The gate potentials of the transfer transistor 103 and reset transistor 104 are appropriately set to the resistance R between the power supply potential VDD and the cathode of the photoelectric conversion element 102. The current flowing through the photoelectric conversion element 102 when it is in an avalanche breakdown state, along with the resistance R, allows for the detection of photon incidence as a voltage change. By counting this detected voltage change, the number of incident photons can be counted, enabling the acquisition of high signal-to-noise ratio images for subjects with low brightness. The mode in which the device is operated with switch S10 ON and switch S11 OFF is defined as APD mode.

[0067] In APD mode, a counter is required for the photon counting operation. For example, the pixel signal processing control unit 711 is configured to have a counter function. The count value held in the pixel signal processing control unit 711 is read out from the drive timing control unit 700 by a pixel selection operation via the control line 703. The selected count value is read out from wiring 751, control unit 750, and wiring 723 to the digital signal processing unit 720 and digital signal processing control unit 730.

[0068] On the other hand, when switch S10 is off and switch S11 is on, as described in the first embodiment, the photoelectric conversion element 102 accumulates photocharge through an accumulation operation for a specific period of time, and an image is obtained by reading out the photocharge. This operation mode is called the PD mode.

[0069] In this embodiment, as in the first and second embodiments, pixels are controlled by the pixel signal processing control unit 711. Specifically, in this embodiment, the APD mode and PD mode are controlled by signal processing by the pixel signal processing control unit 711 and the digital signal processing control unit 730. In this embodiment, the path through which the pixel signal is input to the digital signal processing unit 720 differs between the APD mode and the PD mode. For processing in APD mode and PD mode, information is input from the pixel signal processing control unit 711 to the digital signal processing unit 720 via wiring 751, control unit 705, and wiring 723, and appropriate digital signal processing and rearrangement are performed.

[0070] Generally, in regions with low subject brightness, the signal-to-noise ratio (SNR) is more favorable in APD mode compared to PD mode. In regions where the subject brightness is above a certain level, the difference in SNR between APD mode and PD mode is reduced due to the effect of optical shot noise. On the other hand, as the subject brightness increases, the number of photon count operations increases in APD mode, resulting in higher power consumption. Therefore, in regions where the subject brightness is above a certain level, operating in PD mode is more efficient from a power consumption perspective.

[0071] In this embodiment, the pixel signal processing control unit 711 and the digital signal processing control unit 730 can switch the drive of the pixel 101 between APD mode and PD mode according to the brightness of the subject, thereby enabling pixels with a high signal-to-noise ratio while suppressing an increase in power consumption.

[0072] (Fourth Embodiment) Figure 11 is a block diagram showing the configuration of the photoelectric conversion system 11200 according to this embodiment. The photoelectric conversion system 11200 of this embodiment includes a photoelectric conversion device 11204. Here, the photoelectric conversion device 11204 can be any of the photoelectric conversion devices described in the above embodiments. The photoelectric conversion system 11200 can be used, for example, as an imaging system. Specific examples of imaging systems include digital still cameras, digital camcorders, surveillance cameras, network cameras, etc. In Figure 11, an example of a digital still camera is shown as the photoelectric conversion system 11200.

[0073] The photoelectric conversion system 11200 shown in Figure 11 includes a photoelectric converter 11204 and a lens 11202 that forms an optical image of the subject onto the photoelectric converter 11204. The photoelectric conversion system 11200 also includes an aperture 11203 for varying the amount of light passing through the lens 11202 and a barrier 11201 for protecting the lens 11202. The lens 11202 and aperture 11203 form an optical system that focuses light onto the photoelectric converter 11204.

[0074] The photoelectric conversion system 11200 includes a signal processing unit 11205 that processes the output signal output from the photoelectric conversion device 11204. The signal processing unit 11205 performs signal processing operations that perform various corrections and compressions on the input signal as needed before outputting it. The photoelectric conversion system 11200 further includes a buffer memory unit 11206 for temporarily storing image data, and an external interface unit (external I / F unit) 11209 for communicating with an external computer or the like. Furthermore, the photoelectric conversion system 11200 includes a recording medium 11211 such as a semiconductor memory for recording or reading imaging data, and a recording medium control interface unit (recording medium control I / F unit) 11210 for recording or reading from the recording medium 11211. The recording medium 11211 may be built into the photoelectric conversion system 11200 or may be detachable. In addition, communication between the recording medium control I / F unit 11210 and the recording medium 11211, and communication from the external I / F unit 11209 may be performed wirelessly.

[0075] Furthermore, the photoelectric conversion system 11200 includes an overall control and calculation unit 11208 that performs various calculations and controls the entire digital still camera, and a timing generation unit 11207 that outputs various timing signals to the photoelectric conversion device 11204 and the signal processing unit 11205. Here, the timing signals and the like may be input from an external source, and the photoelectric conversion system 11200 only needs to include at least the photoelectric conversion device 11204 and the signal processing unit 11205 that processes the output signals output from the photoelectric conversion device 11204. The overall control and calculation unit 11208 and the timing generation unit 11207 may be configured to perform some or all of the control functions of the photoelectric conversion device 11204.

[0076] The photoelectric converter 11204 outputs an image signal to the signal processing unit 11205. The signal processing unit 11205 performs predetermined signal processing on the image signal output from the photoelectric converter 11204 and outputs image data. The signal processing unit 11205 also generates an image using the image signal. The signal processing unit 11205 may also perform distance measurement calculations on the signal output from the photoelectric converter 11204. The signal processing unit 11205 and the timing generation unit 11207 may be mounted on the photoelectric converter. In other words, the signal processing unit 11205 and the timing generation unit 11207 may be provided on the substrate on which the pixels are arranged, or they may be provided on a separate substrate. By configuring an imaging system using the photoelectric converters of each embodiment described above, an imaging system capable of acquiring higher quality images can be realized.

[0077] (Fifth embodiment) Figure 12 is a block diagram showing a photoelectric conversion system using a photoelectric conversion device described in any one of the first to third embodiments. Figure 12 shows an example configuration of a distance image sensor as a photoelectric conversion system.

[0078] As shown in Figure 12, the distance image sensor 12401 is configured to include an optical system 12407, a photoelectric converter 12408, an image processing circuit 12404, a monitor 12405, and a memory 12406. The distance image sensor 12401 receives light (modulated light or pulsed light) that is projected from a light source device 12409 toward the subject and reflected from the surface of the subject, thereby acquiring a distance image corresponding to the distance to the subject.

[0079] The optical system 12407 is composed of one or more lenses and guides the image light (incident light) from the subject to the photoelectric converter 12408, where it forms an image on the light-receiving surface (sensor part) of the photoelectric converter 12408.

[0080] The photoelectric converter 12408 is one of the photoelectric converters from each of the embodiments described above, and a distance signal indicating the distance obtained from the received light signal output from the photoelectric converter 12408 is supplied to the image processing circuit 12404.

[0081] The image processing circuit 12404 performs image processing to construct a distance image based on the distance signal supplied from the photoelectric converter 12408. The distance image (image data) obtained through this image processing is then supplied to the monitor 12405 for display or supplied to the memory 406 for storage (recording).

[0082] With the distance image sensor 12401 configured in this way, by applying the photoelectric conversion device described above, the characteristics of the pixels are improved, and for example, more accurate distance images can be acquired.

[0083] (Sixth Embodiment) The photoelectric conversion system according to the sixth embodiment is a photoelectric conversion system that utilizes the photoelectric conversion device described in any one of the first to third embodiments. Figure 13 shows a schematic example of the configuration of an endoscopic surgical system, which is an example of a photoelectric conversion system.

[0084] Figure 13 illustrates a surgeon (physician) 13131 performing surgery on a patient 13132 on a patient bed 13133 using an endoscopic surgical system 13003. As shown in the figure, the endoscopic surgical system 13003 consists of an endoscope 13100, surgical instruments 13110, and a cart 13134 equipped with various devices for endoscopic surgery.

[0085] The endoscope 13100 consists of a barrel 13101, the tip of which is inserted into the body cavity of the patient 13132 for a predetermined length, and a camera head 13102 connected to the proximal end of the barrel 13101. In the illustrated example, the endoscope 13100 is shown as a so-called rigid endoscope having a rigid barrel 13101, but the endoscope 13100 may also be configured as a so-called flexible endoscope having a flexible barrel.

[0086] An opening into which an objective lens is fitted is provided at the tip of the endoscope tube 13101. A light source device 13203 is connected to the endoscope 13100, and the light generated by the light source device 13203 is guided to the tip of the endoscope tube by a light guide extending inside the endoscope tube 13101. This light is shone through the objective lens towards the object to be observed inside the body cavity of the patient 13132. The endoscope 13100 may be a straight-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0087] The camera head 13102 contains an optical system and a photoelectric converter. Reflected light from the object being observed (observation light) is focused by the optical system into the photoelectric converter. The photoelectric converter converts the observation light into electrical signals, generating an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observed image. The photoelectric converter can be any of the photoelectric converters described in the embodiments described above. The image signal is transmitted as RAW data to the camera control unit (CCU) 13135.

[0088] The CCU13135 consists of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and other components, and comprehensively controls the operation of the endoscope 13100 and the display device 13136. Furthermore, the CCU13135 receives an image signal from the camera head 13102 and performs various image processing operations on that image signal, such as development processing (demosaic processing), to display the image based on that image signal.

[0089] The display device 13136 displays an image based on an image signal that has been processed by the CCU 13135, under control from the CCU 13135.

[0090] The light source device 13203 consists of a light source such as an LED (Light Emitting Diode) and supplies illumination light to the endoscope 13100 when photographing the surgical area, etc.

[0091] The input device 13137 is an input interface for the endoscopic surgical system 13003. The user can input various types of information and instructions to the endoscopic surgical system 13003 via the input device 13137.

[0092] The treatment instrument control device 13138 controls the drive of the energy treatment instrument 13112 for purposes such as tissue cauterization, incision, or blood vessel sealing.

[0093] The light source device 13203, which supplies illumination light to the endoscope 13100 when photographing the surgical area, can be composed of, for example, an LED, a laser light source, or a combination thereof. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so the white balance of the captured image can be adjusted in the light source device 13203. In this case, it is also possible to capture images corresponding to each of the RGB colors in time-division by irradiating the observation target with laser light from each of the RGB laser light sources in time-division and controlling the drive of the image sensor of the camera head 13102 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter on the image sensor.

[0094] Furthermore, the light source device 13203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. By controlling the drive of the image sensor of the camera head 13102 in synchronization with the timing of the change in light intensity, images can be acquired in time-division order, and these images can be combined to generate high dynamic range images without so-called black crushing and white clipping.

[0095] Furthermore, the light source device 13203 may be configured to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, the wavelength dependence of light absorption in body tissue is utilized. Specifically, by irradiating with narrowband light compared to the irradiation light used during normal observation (i.e., white light), predetermined tissues such as blood vessels on the surface of mucosa can be imaged with high contrast. Alternatively, in special light observation, fluorescence observation may be performed to obtain an image from fluorescence generated by irradiation with excitation light. In fluorescence observation, excitation light can be irradiated onto body tissue and fluorescence from the body tissue can be observed, or a reagent such as indocyanine green (ICG) can be injected into body tissue and excitation light corresponding to the fluorescence wavelength of the reagent can be irradiated onto the body tissue to obtain a fluorescence image. The light source device 13203 may be configured to supply narrowband light and / or excitation light corresponding to such special light observation.

[0096] (Seventh Embodiment) The photoelectric conversion system and mobile unit of this embodiment will be described with reference to Figure 14. Figure 14 is a schematic diagram showing an example of the configuration of the photoelectric conversion system and mobile unit according to this embodiment. In this embodiment, an example of an in-vehicle camera is shown as the photoelectric conversion system.

[0097] Figure 14 shows an example of a vehicle system and a photoelectric conversion system mounted thereon for imaging. The photoelectric conversion system 14301 includes a photoelectric converter 14302, an image preprocessing unit 14315, an integrated circuit 14303, and an optical system 14314. The optical system 14314 forms an optical image of the subject on the photoelectric converter 14302. The photoelectric converter 14302 converts the optical image of the subject formed by the optical system 14314 into an electrical signal. The photoelectric converter 14302 is one of the photoelectric converters in each of the embodiments described above. The image preprocessing unit 14315 performs predetermined signal processing on the signal output from the photoelectric converter 14302. The functions of the image preprocessing unit 14315 may be incorporated into the photoelectric converter 14302. The photoelectric conversion system 14301 is provided with at least two sets of optical systems 14314, photoelectric conversion devices 14302, and image preprocessing units 14315, and the output from each set of image preprocessing units 14315 is input to the integrated circuit 14303.

[0098] The integrated circuit 14303 is an integrated circuit for imaging system applications and includes an image processing unit 14304 with memory 14305, an optical distance measuring unit 14306, a distance measurement calculation unit 14307, an object recognition unit 14308, and an anomaly detection unit 14309. The image processing unit 14304 performs image processing such as development and defect correction on the output signal of the image preprocessing unit 14315. The memory 14305 stores the primary storage of the captured image and the location of defects in the captured pixels. The optical distance measuring unit 14306 focuses on the subject and measures the distance. The distance measurement calculation unit 14307 calculates distance measurement information from multiple image data acquired by multiple photoelectric converters 14302. The object recognition unit 14308 recognizes subjects such as cars, roads, signs, and people. When the anomaly detection unit 14309 detects an anomaly in the photoelectric converter 14302, it alerts the main control unit 14313 to the anomaly.

[0099] The integrated circuit 14303 may be implemented by specially designed hardware, by a software module, or by a combination of these. It may also be implemented by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination of these.

[0100] The main control unit 14313 oversees and controls the operation of the photoelectric conversion system 14301, the vehicle sensor 14310, the control unit 14320, etc. Alternatively, there may be no main control unit 14313, and the photoelectric conversion system 14301, the vehicle sensor 14310, and the control unit 14320 may each have their own communication interfaces and send and receive control signals via a communication network (e.g., CAN standard).

[0101] The integrated circuit 14303 has the function of receiving control signals from the main control unit 14313 or transmitting control signals and set values ​​to the photoelectric converter 14302 via its own control unit.

[0102] The photoelectric conversion system 14301 is connected to the vehicle sensor 14310 and can detect the vehicle's driving conditions, such as vehicle speed, yaw rate, and steering angle, as well as the external environment and the state of other vehicles and obstacles. The vehicle sensor 14310 also serves as a distance information acquisition means for acquiring distance information to objects. Furthermore, the photoelectric conversion system 14301 is connected to the driver assistance control unit 1311, which performs various driving assistance functions such as automatic steering, automatic cruising, and collision avoidance. In particular, regarding the collision judgment function, it determines whether a collision with another vehicle or obstacle has occurred and estimates the collision based on the detection results of the photoelectric conversion system 14301 and the vehicle sensor 14310. This enables avoidance control when a collision is estimated and activation of safety devices in the event of a collision.

[0103] Furthermore, the photoelectric conversion system 14301 is also connected to a warning device 14312 that issues a warning to the driver based on the judgment result of the collision judgment unit. For example, if the collision judgment unit determines that there is a high probability of collision, the main control unit 14313 performs vehicle control to avoid a collision or mitigate damage by applying the brakes, releasing the accelerator, or suppressing engine output. The warning device 14312 warns the user by sounding an alarm, displaying warning information on a display screen such as the car navigation system or instrument panel, or vibrating the seat belt or steering wheel.

[0104] In this embodiment, the photoelectric conversion system 14301 captures images of the area around the vehicle, for example, the front or rear. Figure 14(b) shows an example of the arrangement of the photoelectric conversion system 14301 when it captures images of the area in front of the vehicle.

[0105] The two photoelectric converters 14302 are positioned in front of the vehicle 14300. Specifically, the center line of the vehicle 14300 with respect to its direction of movement or external shape (e.g., vehicle width) is considered as the axis of symmetry, and the two photoelectric converters 1302 are positioned symmetrically with respect to that axis of symmetry. This configuration is preferable for acquiring distance information between the vehicle 14300 and the object being photographed and for determining the possibility of collision. Furthermore, it is preferable that the photoelectric converters 14302 are positioned so as not to obstruct the driver's field of view when the driver is viewing the situation outside the vehicle 14300 from the driver's seat. The warning device 14312 is preferably positioned so as to be easily visible to the driver.

[0106] Furthermore, although this embodiment describes control to avoid collisions with other vehicles, it can also be applied to control that automatically follows other vehicles or control that automatically drives without deviating from the lane. In addition, the photoelectric conversion system 14301 can be applied not only to vehicles such as the vehicle itself, but also to mobile objects (mobile devices) such as ships, aircraft, or industrial robots. Moreover, it can be applied not only to mobile objects, but also to a wide range of devices that utilize object recognition, such as intelligent transportation systems (ITS).

[0107] The photoelectric conversion device of the present invention may further be configured to acquire various types of information, such as distance information.

[0108] (Eighth embodiment) Figure 15 shows a pair of glasses 16600 (smart glasses), which is an example of a photoelectric conversion system according to this embodiment. The glasses 16600 have a photoelectric conversion device 16602. The photoelectric conversion device 16602 is a photoelectric conversion device described in any one of the first to third descriptions. In addition, a display device including a light-emitting device such as an OLED or LED may be provided on the back side of the lens 16601. There may be one or more photoelectric conversion devices 16602. In addition, multiple types of photoelectric conversion devices may be used in combination. The arrangement position of the photoelectric conversion device 16602 is not limited to that shown in Figure 15(a).

[0109] The eyeglasses 16600 further include a control device 16603. The control device 16603 functions as a power source that supplies power to the photoelectric converter 16602 and the display device. The control device 16603 also controls the operation of the photoelectric converter 16602 and the display device. The lens 16601 has an optical system formed therein for focusing light onto the photoelectric converter 16602.

[0110] Figure 15(b) illustrates a pair of glasses 16610 (smart glasses) according to one application example. The glasses 16610 have a control device 16612, which is equipped with a photoelectric converter equivalent to a photoelectric converter 16602 and a display device. The lens 16611 has an optical system formed therein for projecting light emitted from the photoelectric converter in the control device 16612 and from the display device, and an image is projected onto the lens 16611. The control device 16612 functions as a power supply that provides power to the photoelectric converter and the display device, and also controls the operation of the photoelectric converter and the display device. The control device may have a gaze detection unit that detects the wearer's gaze. Gaze detection may use infrared light. The infrared light emitter emits infrared light towards the eyeball of the user who is gazing at the displayed image. An imaging unit having a photodetector detects the reflected light from the eyeball of the emitted infrared light, thereby obtaining an image of the eyeball. By having a reduction mechanism that reduces the amount of light transmitted from the infrared light-emitting part to the display part in a planar view, the degradation of image quality is reduced.

[0111] The user's gaze towards a displayed image is detected from an image of the eyeball obtained by imaging with infrared light. Any known method can be applied to gaze detection using an image of the eyeball. As an example, a gaze detection method based on the Purkinje image obtained by the reflection of the irradiated light from the cornea can be used.

[0112] More specifically, gaze detection processing is performed based on the pupil-corneal reflection method. Using the pupil-corneal reflection method, a gaze vector representing the orientation (rotation angle) of the eyeball is calculated based on the pupil image and Purkinje image contained in the captured image of the eyeball, thereby detecting the user's gaze.

[0113] The display device of this embodiment includes a photoelectric converter having a light-receiving element, and may control the display image of the display device based on the user's gaze information from the photoelectric converter.

[0114] Specifically, the display device determines a first field of view that the user is fixated on, and a second field of view other than the first field of view, based on gaze information. The first and second field of view may be determined by the control device of the display device, or they may be determined by an external control device and received by the display device. Within the display area of ​​the display device, the display resolution of the first field of view may be controlled to be higher than the display resolution of the second field of view. In other words, the resolution of the second field of view may be lower than that of the first field of view.

[0115] Furthermore, the display area has a first display area and a second display area different from the first display area, and a higher priority area may be determined from the first and second display areas based on gaze information. The first and second view areas may be determined by the control device of the display device, or they may be determined by an external control device and received. The resolution of the higher priority area may be controlled to be higher than the resolution of the areas other than the higher priority area. In other words, the resolution of areas with relatively lower priority may be set lower.

[0116] AI may be used to determine the first field of view area and high-priority areas. The AI ​​may be a model configured to estimate the angle of gaze and the distance to the target object at the end of the line of sight from the image of the eye, using the image of the eye and the direction the eye was actually looking in that image as training data. The AI ​​program may be installed in the display device, the photoelectric converter, or an external device. If installed in an external device, it will be transmitted to the display device via communication.

[0117] When display control is based on visual detection, this method is preferably applicable to smart glasses that further include a photoelectric converter for capturing images of the surrounding environment. The smart glasses can display the captured external information in real time.

[0118] (Ninth Embodiment) The photoelectric conversion system of this embodiment will be described with reference to Figure 16. Figure 16 shows a diagnostic support system, which is an example of a photoelectric conversion system. It can be applied to a pathological diagnostic system in which a physician or other medical professional observes cells or tissues collected from a patient to diagnose a lesion, or to a diagnostic support system that assists in such a diagnosis. The system of this embodiment may diagnose or assist in diagnosing a lesion based on acquired images.

[0119] As shown in Figure 16, the system of this embodiment includes one or more pathology systems 15510. It may also include an analysis unit 15530 and a medical information system 15540.

[0120] Each of the one or more pathology systems 15510 is primarily used by pathologists and is installed, for example, in research laboratories or hospitals. Each pathology system 15510 may be installed in different hospitals and is connected to the analysis unit 15530 and the medical information system 15540 via various networks such as wide area networks and local area networks.

[0121] Each pathology system 15510 includes a microscope 15511, a server 15512, and a display device 15513.

[0122] Microscope 15511 has the functions of an optical microscope and captures images of objects placed on glass slides, acquiring pathological images as digital images. The objects of observation may be, for example, tissues or cells taken from a patient, and may include organ tissue, saliva, blood, etc.

[0123] Server 15512 stores pathological images acquired by microscope 15511 in a memory unit (not shown). Furthermore, when server 15512 receives a viewing request, it can search for pathological images stored in memory, etc., and display the retrieved pathological images on display device 15513. Server 15512 and display device 15513 may be connected via a display control device or the like.

[0124] In this case, if the object of observation is a solid object such as a piece of organ tissue, this object of observation may be, for example, a stained section. The section may be prepared, for example, by slicing a block piece cut from a specimen such as an organ. The block piece may also be fixed with paraffin or the like during slicing.

[0125] The microscope 15511 may include a low-resolution imaging unit for imaging at low resolution and a high-resolution imaging unit for imaging at high resolution. The low-resolution imaging unit and the high-resolution imaging unit may have different optical systems or may have the same optical system. If they have the same optical system, the resolution of the microscope 15511 may be changed depending on the object being imaged.

[0126] The object to be observed is placed on a glass slide or the like and positioned on a stage located within the field of view of the microscope 15511. First, the microscope 15511 acquires an overall image of the field of view using a low-resolution imaging unit and identifies the area of ​​the object to be observed from the acquired overall image. Next, the microscope 15511 divides the area where the object to be observed is located into multiple divided regions of a predetermined size and acquires high-resolution images of each divided region by sequentially imaging each divided region with a high-resolution imaging unit. When switching between the target divided regions, the stage may be moved, the imaging optical system may be moved, or both may be moved. In addition, each divided region may overlap with adjacent divided regions to prevent the occurrence of areas that are not captured due to unintended slippage of the glass slide. Furthermore, the overall image may include identification information to associate the overall image with the patient. This identification information may be, for example, a string of characters or a QR code (registered trademark).

[0127] High-resolution images acquired by microscope 15511 are input to server 15512. Server 15512 can divide each high-resolution image into smaller sub-images. Once sub-images are generated, server 15512 performs a synthesis process on all sub-images, combining a predetermined number of adjacent sub-images to generate a single image. This synthesis process can be repeated until a single sub-image is finally generated. This process generates a pyramidal structure of sub-images, where each layer consists of one or more sub-images. In this pyramidal structure, the number of pixels in a sub-image from one layer is the same as that of a sub-image from a different layer, but their resolutions are different. For example, when combining four 2x2 sub-images to generate one sub-image from the upper layer, the resolution of the upper layer sub-image is half the resolution of the lower layer sub-images used in the synthesis.

[0128] By constructing a pyramidal structure of partial images, it becomes possible to switch the level of detail displayed on the display device depending on the hierarchical level to which the tile image being displayed belongs. For example, when the lowest-level partial image is used, a narrow area of ​​the object can be displayed in detail, while as higher-level partial images are used, a wider area of ​​the object can be displayed coarser.

[0129] The generated pyramidal structure sub-images can be stored, for example, in memory. When server 15512 receives a request from another device (for example, analysis unit 15530) to acquire a sub-image containing identification information, it transmits the sub-image corresponding to the identification information to the other device.

[0130] Furthermore, partial images, which are pathological images, may be generated for each imaging condition, such as focal length and staining conditions. When partial images are generated for each imaging condition, other pathological images corresponding to different imaging conditions but in the same region as the specific pathological image may be displayed side by side with the specific pathological image. The specific imaging conditions may be specified by the viewer. In addition, if multiple imaging conditions are specified by the viewer, pathological images of the same region corresponding to each imaging condition may be displayed side by side.

[0131] Furthermore, server 15512 may store the pyramidal partial image sets in another storage device, such as a cloud server. Moreover, some or all of the partial image generation process described above may be performed on a cloud server. By using these partial images, the user can gain the sensation of observing an object while changing the magnification. In other words, by controlling the display, it can function like a virtual microscope. Here, the virtual magnification actually corresponds to the resolution.

[0132] The medical information system 15540 is a so-called electronic medical record system that stores information related to diagnosis, such as patient identification information, patient disease information, test information and image information used for diagnosis, diagnosis results, and prescribed medications. For example, a pathological image obtained by imaging an object of observation of a patient can be stored via the server 15512 and then displayed on the display device 15514. A pathologist using the pathology system 15510 makes a pathological diagnosis based on the pathological image displayed on the display device 15513. The pathological diagnosis results made by the pathologist are stored in the medical information system 15540.

[0133] The analysis unit 15530 can perform analysis on pathological images. This analysis can utilize a learning model created by machine learning. The analysis unit 15530 may derive classification results for specific regions, tissue identification results, etc., as analysis results. Furthermore, the analysis unit 15530 may derive identification results for cell information, number, location, brightness information, etc., and scoring information for these. This information obtained by the analysis unit 15530 may be displayed on the display device 15513 of the pathology system 15510 as diagnostic support information.

[0134] The analysis unit 15530 may be a server system consisting of one or more servers (including cloud servers). Alternatively, the analysis unit 15530 may be integrated into, for example, server 15512 within the pathology system 15510. In other words, various analyses of pathology images may be performed within the pathology system 15510.

[0135] The photoelectric conversion device described in the first to third embodiments above can be suitably applied to, for example, the microscope 15511 among the configurations described above. Specifically, it can be applied to the low-resolution imaging unit and / or high-resolution imaging unit of the microscope 15511. This makes it possible to miniaturize the low-resolution imaging unit and / or high-resolution imaging unit, and consequently, the microscope 15511. As a result, the transport of the microscope 15511 becomes easier, making it possible to facilitate system introduction and system reconfiguration. Furthermore, by applying the photoelectric conversion device described in the embodiments above, some or all of the processing from pathological image acquisition to pathological image analysis can be performed on the fly within the microscope 15511, thus enabling the output of faster and more accurate diagnostic support information.

[0136] The configuration described above can be applied not only to diagnostic support systems but also to biological microscopes in general, such as confocal microscopes, fluorescence microscopes, and video microscopes. Here, the object of observation may be biological samples such as cultured cells, fertilized eggs, and sperm, biomaterials such as cell sheets and three-dimensional cell tissues, or living organisms such as zebrafish and mice. Furthermore, the object of observation is not limited to glass slides; it can also be observed while stored in well plates, petri dishes, etc.

[0137] Furthermore, moving images may be generated from still images of the object being observed using a microscope. For example, moving images may be generated from still images taken continuously over a predetermined period, or image sequences may be generated from still images taken at predetermined intervals. By generating moving images from still images in this way, it becomes possible to analyze the dynamic characteristics of the object being observed, such as the pulsation, extension, and migration of cancer cells, nerve cells, myocardial tissue, and sperm, as well as the division process of cultured cells and fertilized eggs, using machine learning.

[0138] [Modified Embodiment] The present invention is not limited to the embodiments described above and can be modified in various ways. For example, examples in which a part of the configuration of one embodiment is added to another embodiment, or in which a part of the configuration of another embodiment is replaced, are also included as embodiments of the present invention. It should be noted that the above embodiments are merely examples of concrete implementations of the present invention, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features. [Explanation of symbols]

[0139] 10. First board 20 Second board 100 pixel section 500 Signal Processing Unit 520, 740 memory section 700 Drive Timing Processing Unit 710, 750 Control Unit 711 Pixel signal processing control unit 730 Digital signal processing control unit 720 Digital signal processing unit

Claims

1. a first substrate having pixels including photoelectric conversion elements; a second substrate stacked on the first substrate, the second substrate having a first control unit including a first signal processing unit that processes signals from the pixels; the signal from the pixel is output to a second signal processing unit disposed at a position different from the first signal processing unit, a path through which a signal from the pixel is output to the first signal processing unit is different from a path through which a signal from the pixel is output to the second signal processing unit, The photoelectric conversion device, wherein the first control unit controls the pixels based on the signals processed by the first signal processing unit.

2. 2. The photoelectric conversion device according to claim 1, wherein the first control unit controls the second signal processing unit.

3. the second signal processing unit has an amplifier; 3. The photoelectric conversion device according to claim 2, wherein the first control unit controls a voltage amplification factor in the processing in the second signal processing unit.

4. the second signal processing unit has an analog-to-digital conversion circuit, 4. The photoelectric conversion device according to claim 2, wherein the first control unit controls an analog-to-digital conversion gain of the analog-to-digital conversion circuit in the processing in the second signal processing unit.

5. the pixel includes a floating diffusion region; 5. The photoelectric conversion device according to claim 1, wherein the first control unit switches a capacitance value of a floating diffusion region of the pixel.

6. The photoelectric conversion device according to any one of claims 1 to 5, characterized in that the first control unit controls at least one of the pixel and the second signal processing unit before signal processing of the signal from the pixel is started in the second signal processing unit.

7. The photoelectric conversion device according to claim 6, wherein the first control unit controls at least one of the pixel and the second signal processing unit before the signal from the pixel is input to the second signal processing unit.

8. 8. The photoelectric conversion device according to claim 1, wherein the second signal processing unit is disposed on the second substrate.

9. 8. The photoelectric conversion device according to claim 1, wherein the second signal processing unit is disposed on the first substrate.

10. A photoelectric conversion device according to any one of claims 1 to 9, further comprising a second control unit that controls at least one of the pixel, the first control unit, and the second signal processing unit based on the signal processed by the second signal processing unit.

11. 11. The photoelectric conversion device according to claim 10, wherein at least one of the first control unit and the second control unit controls an accumulation time of the pixel.

12. the first substrate has a pixel portion including a plurality of the pixels, the pixel unit has pixel blocks in which the plurality of pixels are divided into blocks of a plurality of pixels, the first control unit is arranged two-dimensionally in a plan view, 12. The photoelectric conversion device according to claim 1, wherein the pixel block and the first control unit are arranged to overlap each other in an orthogonal projection onto the first surface of the first substrate.

13. 13. The photoelectric conversion device according to claim 1, wherein at least one of the first substrate and the second substrate has a processing unit that executes processing based on a neural network calculation model.

14. the pixel includes the photoelectric conversion element and a second photoelectric conversion element, the photoelectric conversion element and the second photoelectric conversion element have different sensitivities, 14. The photoelectric conversion device according to claim 1, wherein readout of the photoelectric conversion element and the second photoelectric conversion element is controlled by the first control unit or the second control unit.

15. the pixel includes a first mode in which a reverse bias voltage that causes avalanche multiplication is supplied to a photoelectric conversion element, and a second mode in which a voltage that does not cause avalanche multiplication is supplied to the photoelectric conversion element, 14. The photoelectric conversion device according to claim 1, wherein the switching between the first mode and the second mode is performed by the first control unit or the second control unit.

16. The photoelectric conversion device according to any one of claims 1 to 15, a signal processing unit that processes a signal output from the photoelectric conversion device.

17. The photoelectric conversion device according to any one of claims 1 to 15, a distance information acquisition means for acquiring distance information to an object from distance measurement information based on a signal from the photoelectric conversion device, A moving body further comprising a control means for controlling the moving body based on the distance information.

Images