Solid-state imaging device and electronic apparatus

Abstract

[Problem] To provide a solid-state imaging device and an electronic apparatus that can expand a dynamic range with a simple control by setting exposure times in a plurality of patterns for each pixel. [Solution] A solid-state imaging device according to a first aspect of the present disclosure comprises: a pixel array unit that has a plurality of pixels arranged in a two-dimensional array; a vertical drive unit that supplies a first drive signal to the plurality of pixels in the vertical direction; a horizontal drive unit that supplies a second drive signal to the plurality of pixels in the horizontal direction; and a control unit that controls the horizontal drive unit and the vertical drive unit. Each of the plurality of pixels includes an access transistor that controls the amount of charge accumulated in the pixel on the basis of the first drive signal and the second drive signal. The control unit controls the vertical drive unit and the horizontal drive unit so as to output the first drive signal and the second drive signal at irregular timings.

Description

Solid-state imaging device and electronic device

[0001] The present invention relates to a solid-state imaging device and an electronic device.

[0002] 2. Description of the Related Art In solid-state imaging devices such as CMOS image sensors, it is desirable to widen the dynamic range of captured images. There is also a technique for widening the dynamic range by changing the exposure time of pixels included in the solid-state imaging device.

[0003] Japanese Patent Application Laid-Open No. 2020-80521

[0004] In the above-described solid-state imaging device, it is possible to change the exposure time for each pixel block made up of a plurality of pixels, and it is possible to expand the dynamic range of the captured image.

[0005] However, the exposure time can only be controlled in units of pixel blocks, such as 3×4 pixels, and providing a complex pixel control unit for each pixel block is not practical because it increases the circuit scale.

[0006] In view of these problems, the present disclosure provides a solid-state imaging device and electronic equipment that can set a plurality of exposure time patterns for each pixel and expand the dynamic range with simple control.

[0007] According to a first aspect of the present disclosure, there is provided a solid-state imaging device including a pixel array unit in which a plurality of pixels are arranged in a two-dimensional array, a vertical drive unit that supplies a first drive signal to the plurality of pixels in the vertical direction, a horizontal drive unit that supplies a second drive signal to the plurality of pixels in the horizontal direction, and a control unit that controls the horizontal drive unit and the vertical drive unit, wherein each of the plurality of pixels includes an access transistor that controls the amount of charge stored in the pixel based on the first drive signal and the second drive signal, and the control unit controls the vertical drive unit and the horizontal drive unit to output the first drive signal and the second drive signal at irregular timing. As a result, for example, by turning on the access transistor under the control of the control unit, the solid-state imaging device can set a plurality of exposure time patterns for each pixel without setting each row or column to the same state, thereby achieving a high dynamic range.

[0008] In this first aspect, the access transistor includes a first gate that receives the first drive signal and a second gate that receives the second drive signal, and an N-type channel layer connecting the first gate and the second gate has a lower concentration than an N-type diffusion region connected to a power supply voltage. The access transistor initializes the charge stored in the photodiode based on the first drive signal and the second drive signal. This eliminates the need for a deep N-well region in the solid-state imaging device, thereby reducing the circuit area.

[0009] In this first aspect, each of the plurality of pixels further includes a first floating diffusion that accumulates charge transferred from a photodiode and a second floating diffusion that accumulates charge overflowing from the first floating diffusion. As a result, for example, the solid-state imaging device can temporarily accumulate charge accumulated in the photodiode in the first floating diffusion by turning on the transfer transistor under control of the control unit, and can also accumulate charge overflowing from the first floating diffusion in the second floating diffusion by turning on the connection transistor.

[0010] In this first aspect, one of the source and drain of the access transistor is connected to the power supply voltage, and the other of the source and drain is connected to the photodiode. As a result, for example, the solid-state imaging device can randomly turn on the access transistor under the control of a control unit, thereby achieving a high dynamic range with simple control.

[0011] In this first aspect, one of the source and drain of the access transistor is connected to the first floating diffusion, and the other of the source and drain is connected to the photodiode. As a result, for example, the solid-state imaging device can initialize all charges in the photodiode and the first and second floating diffusions, or can selectively initialize only the first and second floating diffusions, by controlling each transistor with a control unit.

[0012] In this first aspect, the access transistor initializes the charge accumulated in the photodiode based on the first drive signal and the second drive signal. As a result, for example, the solid-state imaging device can randomly turn on the access transistor under the control of a control unit, thereby achieving a high dynamic range with simple control.

[0013] In this first aspect, each of the plurality of pixels further includes a drain transistor that connects a photodiode to a power supply voltage based on a third drive signal supplied from the access transistor. As a result, for example, the solid-state imaging device can randomly turn on the access transistors under the control of the control unit, thereby achieving a high dynamic range with simple control.

[0014] In this first aspect, the access transistor has a gate connected to the vertical drive unit, one of a source and a drain connected to the horizontal drive unit, and the other of the source and drain connected to the gate of the discharge transistor, and outputs the third drive signal. As a result, for example, the solid-state imaging device can randomly turn on the access transistor under control of the control unit, thereby achieving a high dynamic range with simple control.

[0015] In this first aspect, each of the plurality of pixels has a well structure that isolates the potential of the access transistor from a region within the pixel, thereby enabling, for example, the solid-state imaging device to suppress leakage current from the gate electrode of the discharge transistor by floating the discharge transistor.

[0016] In the first aspect, the gate of the drain transistor is formed of P-POLY, so that, for example, the solid-state imaging device can suppress leakage current from the gate electrode of the drain transistor by floating the drain transistor.

[0017] In the first aspect, the irregular timing is random timing, and thus, for example, the solid-state imaging device randomly turns on the access transistor under the control of the control unit, thereby achieving a high dynamic range with simple control.

[0018] In this first aspect, each of the plurality of pixels further includes a storage capacitor that stores charge that has overflowed from the first floating diffusion or the second floating diffusion, and the storage capacitor has an MIM structure. This allows, for example, the solid-state imaging device to store the overflowed charge in the storage capacitor while charge is being stored in the photodiode and the first and second floating diffusions.

[0019] In this first aspect, each of the plurality of pixels further includes a connection transistor connecting the first floating diffusion and the second floating diffusion, and one of a source and a drain of the access transistor is connected to the power supply voltage and the other of the source and drain is connected to the second floating diffusion. As a result, for example, in the solid-state imaging device, when the access transistor is turned on, the charge of the second floating diffusion is initialized.

[0020] In this first aspect, each of the plurality of pixels further includes a reset transistor that initializes the charge accumulated in the second floating diffusion, and one of the source and drain of the access transistor is connected to the first floating diffusion and the other of the source and drain is connected to the second floating diffusion. This allows, for example, the solid-state imaging device to switch conversion efficiency by turning on or off the access transistor.

[0021] According to a second aspect of the present disclosure, there is provided an electronic device including a solid-state imaging device, the electronic device including a recognition unit including a learning model that receives an image as an input and outputs a recognition result, the solid-state imaging device including a pixel array unit in which a plurality of pixels are arranged in a two-dimensional array, a vertical drive unit that supplies a first drive signal to the plurality of pixels in the vertical direction, a horizontal drive unit that supplies a second drive signal to the plurality of pixels in the horizontal direction, and a control unit that controls the horizontal drive unit and the vertical drive unit, each of the plurality of pixels including an access transistor that controls an amount of charge stored in the pixel based on the first drive signal and the second drive signal, the control unit controlling the vertical drive unit and the horizontal drive unit to output the first drive signal and the second drive signal at irregular timings. As a result, for example, the control unit can control each transistor in the solid-state imaging device to initialize all charges in the photodiode and the first and second floating diffusions, or to selectively initialize only the first and second floating diffusions.

[0022] In the second aspect, the access transistor includes a first gate that receives the first drive signal and a second gate that receives the second drive signal, and an N-type channel layer connecting the first gate and the second gate has a lower concentration than an N-type diffusion region connected to a power supply voltage. This eliminates the need for a deep N-well region in the solid-state imaging device, thereby reducing the circuit area.

[0023] In the second aspect, one of the source and drain of the access transistor is connected to the power supply voltage, and the other of the source and drain is connected to a photodiode. As a result, for example, in an electronic device, when the access transistor is turned on, the charge of the second floating diffusion is initialized.

[0024] According to a third aspect of the present disclosure, there is provided an electronic device including a solid-state imaging device, the electronic device further including a blur removal unit including a learning model that receives a plurality of captured images as input and outputs a plurality of reconstructed images from which blur has been removed, the solid-state imaging device including a pixel array unit having a plurality of pixels arranged in a two-dimensional array, a vertical drive unit that supplies a first drive signal to the plurality of pixels in the vertical direction, a horizontal drive unit that supplies a second drive signal to the plurality of pixels in the horizontal direction, and a control unit that controls the horizontal drive unit and the vertical drive unit, each of the plurality of pixels including an access transistor that controls the amount of charge stored in the pixel based on the first drive signal and the second drive signal, the control unit controlling the vertical drive unit and the horizontal drive unit to output the first drive signal and the second drive signal at irregular timing. As a result, for example, the electronic device can achieve a high dynamic range by using pixels with different exposure times and output a highly accurate reconstructed image.

[0025] In the third aspect, the access transistor includes a first gate that receives the first drive signal and a second gate that receives the second drive signal, and an N-type channel layer connecting the first gate and the second gate has a lower concentration than an N-type diffusion region connected to a power supply voltage. This eliminates the need for a deep N-well region in an electronic device, thereby reducing the circuit area.

[0026] In the third aspect, one of the source and drain of the access transistor is connected to the power supply voltage, and the other of the source and drain is connected to a photodiode. As a result, for example, in an electronic device, when the access transistor is turned on, the charge of the second floating diffusion is initialized.

[0027] 1 is a block diagram showing an example of the configuration of an imaging device according to a first embodiment; FIG. 2 is a block diagram showing a schematic configuration of a solid-state imaging device according to the first embodiment; FIG. 3 is an example of a circuit diagram of a pixel according to the first embodiment; FIG. 4 is a diagram explaining the relationship between the state of an access transistor HV and the charge accumulation time in each pixel according to the first embodiment; FIG. 5 is a time chart of shutter (SHT) and read (READ) in one vertical period of a pixel according to the first embodiment; FIG. 6 is a diagram explaining the dynamic range of a solid-state imaging device according to the first embodiment; FIG. 7 is a schematic cross-sectional view of the vicinity of a photodiode PD in a pixel according to the first embodiment; FIG. 8 is a potential diagram of a pixel according to the first embodiment; FIG. 9 is an example of a circuit diagram of a pixel according to a first modified example of the first embodiment; FIG. 10 is a diagram explaining the dynamic range of a solid-state imaging device according to the first modified example of the first embodiment; FIG. 11 is an example of a circuit diagram of a pixel according to a second modified example of the first embodiment; FIG. 12 is a diagram showing the relationship between the amount of charge accumulated in the photodiode PD and the floating diffusion FD2 according to the second modified example of the first embodiment; FIG. 13 is an example of a circuit diagram of a pixel according to a third modified example of the first embodiment; FIG. 14 is a schematic cross-sectional view of the vicinity of the photodiode PD in the third modified example of the first embodiment; FIG. 15 is another schematic cross-sectional view of the vicinity of the photodiode PD in the third modified example of the first embodiment; FIG. 10 is an example circuit diagram of a pixel in a first modified example of the second embodiment. FIG. 11 is a diagram showing the relationship between the amount of charge stored in the photodiode PD and the floating diffusion FD2 in the first modified example of the second embodiment. FIG. 12 is an example circuit diagram of a pixel in a second modified example of the second embodiment. FIG. 13 is a configuration example of an imaging device in a third embodiment. FIG. 14 is a schematic diagram for explaining the operation of the imaging device in the third embodiment. FIG. 15 is a block diagram of an example of a second system configuration in the third embodiment. FIG. 16 is a block diagram of an example of a third system configuration in the third embodiment. FIG. 17 is a block diagram showing the hardware configuration of an example of an information processing device applicable to the third system configuration. FIG. 18 is a functional block diagram of an example of an image pickup device in a fourth embodiment. FIG. 19 is a schematic diagram for explaining the operation of the image pickup device in the fourth embodiment. FIG. 19 is a block diagram showing an example of the configuration of a vehicle control system.FIG. 2 is a diagram illustrating an example of a sensing region.

[0028] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and their description will be omitted as appropriate. The drawings are simplified, and components necessary for implementation other than those shown in the drawings are also included as appropriate. Furthermore, when terms such as "first" and "second" are used in this specification or claims, unless otherwise specified, they do not represent any order or importance, but are used to distinguish one configuration from another.

[0029] Additionally, in this disclosure, the terms "equal to or greater than" and "equal to or less than" can be read as "greater than" and "less than," respectively.

[0030] First Embodiment FIG. 1 is a block diagram showing an example of the configuration of an imaging device 2 according to a first embodiment.

[0031] The imaging device 2 is a device that captures an image of an object, and includes an imaging lens 210, a solid-state imaging device 10, a recording unit 240, and an imaging control unit 230. Examples of the imaging device 2 are a digital camera such as an IoT camera, or an electronic device with an imaging function (such as a smartphone or a personal computer).

[0032] The solid-state imaging device 10 captures an image under the control of the imaging control unit 230. The solid-state imaging device 10 supplies the captured image to the recording unit 240 via a signal line 209.

[0033] The imaging lens 210 collects light and guides it to the solid-state imaging device 10. The imaging control unit 230 controls the solid-state imaging device 10 to capture an image. The imaging control unit 230 supplies an imaging control signal including a vertical synchronization signal VSYNC to the solid-state imaging device 10 via, for example, a signal line 139. The recording unit 240 records the captured image.

[0034] Here, the vertical synchronization signal VSYNC is a signal that indicates the timing of imaging, and a periodic signal with a constant frequency (such as 60 Hz) is used as the vertical synchronization signal VSYNC.

[0035] In this embodiment, the imaging device 2 records the captured image in the recording unit 240, but the captured image may also be transmitted to the outside of the imaging device 2. In this case, an external interface for transmitting the captured image to the imaging device 2 is further provided. The imaging device 2 may also display the captured image. In this case, the imaging device 2 is further provided with a display unit.

[0036] FIG. 2 is a block diagram showing a schematic configuration of the solid-state imaging device 10 according to the first embodiment.

[0037] 2 includes a pixel array section 20 in which a plurality of pixels are arranged in a matrix, and a peripheral circuit section therearound, which includes a vertical drive section 12, an AD conversion section 13, a horizontal drive section 14, a control section 15, a signal processing section 16, a data storage section 17, an input / output section 18, and the like.

[0038] Each pixel arranged in a two-dimensional array in the pixel array unit 20 is composed of a photodiode as a photoelectric conversion unit, a plurality of pixel transistors, etc. The plurality of pixel transistors are, for example, MOS transistors such as transfer transistors, amplification transistors, selection transistors, and reset transistors. Each pixel in the pixel array unit 20 has, for example, red (R), green (G), or blue (B) color filters arranged in a Bayer array, and each pixel outputs a pixel signal of either R, G, or B (hereinafter simply referred to as a signal).

[0039] The vertical drive unit 12 is configured, for example, by a shift register, and drives pixels row by row by supplying drive pulses to each pixel of the pixel array unit 20 via pixel drive wiring (not shown). That is, the vertical drive unit 12 sequentially selects and scans each pixel of the pixel array unit 20 row by row in the vertical direction, and outputs pixel signals based on signal charges generated in the photodiode of each pixel according to the amount of incident light to the signal processing unit 16 through vertical signal lines provided in common for each column. The vertical drive unit 12 also outputs drive signals to first gate electrodes of access transistors HV, which will be described later. The drive signals output from the vertical drive unit 12 are an example of first drive signals.

[0040] The AD converter (ADC) 13 performs CDS (Correlated Double Sampling) processing for removing fixed pattern noise specific to pixels and AD conversion on the signal output from the pixel array unit 20 .

[0041] The horizontal drive unit 14 is configured by, for example, a shift register and sequentially outputs horizontal scanning pulses. The horizontal drive unit also outputs a drive signal to a second gate electrode of an access transistor HV (described later). The drive signal output from the horizontal drive unit 14 is an example of a second drive signal.

[0042] The control unit 15 receives a clock signal input from the outside and data instructing the operation mode, etc., and controls the overall operation of the solid-state imaging device 10. For example, the control unit 15 generates a vertical synchronization signal, a horizontal synchronization signal, etc. based on the input clock signal, and supplies them to the vertical drive unit 12, the AD conversion unit 13, the horizontal drive unit 14, etc.

[0043] The signal processing unit 16 performs black level adjustment processing and column variation correction processing on the pixel signals supplied from the AD conversion unit 13, and then performs white balance processing. After the white balance processing, the signal processing unit 16 performs various types of digital signal processing such as demosaic processing as necessary to generate an image and supply it to the input / output unit 18.

[0044] The data storage unit 17 stores information on lenses, sensor color filters, etc., which will be described later. The signal processing unit 16 can store parameters, etc., input from an external image processing device via the input / output unit 18 in the data storage unit 17, and can appropriately select and execute signal processing based on instructions from the external image processing device.

[0045] The input / output unit 18 outputs pixel signals sequentially input from the signal processing unit 16 to an external image processing device, for example, an ISP (Image Signal Processor) at a subsequent stage, etc. The input / output unit 18 also supplies signals and parameters input from the external image processing device to the signal processing unit 16 and the control unit 15.

[0046] The solid-state imaging device 10 is configured as described above, and is, for example, a CMOS image sensor of a so-called column AD system in which CDS processing and AD conversion processing are performed for each pixel column.

[0047] FIG. 3 is an example of a circuit diagram of the pixel 100 according to the first embodiment.

[0048] In this embodiment, the pixel 100 includes a photodiode PD, a transfer transistor TGL, a selection transistor SEL, an amplification transistor AMP, a floating diffusion FD1, a floating diffusion FD2, a connection transistor FDG, a reset transistor RST, and an access transistor HV. The floating diffusion FD1 is an example of a first floating diffusion, and the floating diffusion FD2 is an example of a second floating diffusion.

[0049] The photodiode PD is, for example, a PN junction photodiode. The photodiode PD generates and accumulates electric charges corresponding to the amount of light received. The transfer transistor TGL is an NMOS transistor provided between the photodiode PD and the floating diffusion FD1. A drive signal TGL is applied to the gate electrode of the transfer transistor TGL. That is, when the drive signal TGL becomes a high potential level (hereinafter simply referred to as "on"), the transfer transistor TGL becomes conductive, and the electric charges accumulated in the photodiode PD are transferred to the floating diffusion FD1 via the transfer transistor TGL.

[0050] The reset transistor RST is an NMOS transistor provided between the power supply voltage VDD and the floating diffusion FD2. A drive signal RST is applied to the gate electrode of the reset transistor RST. When the drive signal RST becomes a high potential level, the reset transistor RST becomes conductive. This initializes (resets) the potential of the region where the floating diffusion FD2 and the charge storage section are coupled to the level of the power supply voltage VDD. The floating diffusions FD1 and FD2 are floating diffusion regions capable of holding a predetermined amount of charge. The charge stored in the floating diffusions FD1 and FD2 is converted into a voltage signal and read out.

[0051] The connection transistor FDG is an NMOS transistor provided between the floating diffusion FD2 and the floating diffusion FD1. A drive signal FDG is applied to the gate electrode of the connection transistor FDG. When the drive signal FDG becomes a high potential level, the connection transistor FDG becomes conductive, the floating diffusions FD1 and FD2 are connected, and charge is transferred from the floating diffusion FD1 to the floating diffusion FD2.

[0052] The amplifier transistor AMP is an NMOS transistor whose gate electrode is connected to the floating diffusion FD1 and whose drain electrode is connected to the power supply voltage VDD. The amplifier transistor AMP serves as the input of a readout circuit for reading out the charges held in the floating diffusions FD1 and FD2, i.e., a source follower circuit. In other words, the amplifier transistor AMP has its source electrode connected to a vertical signal line via a selection transistor, thereby constituting a source follower circuit together with a constant current source connected to the vertical signal line.

[0053] The selection transistor SEL is an NMOS transistor provided between the source electrode of the amplification transistor AMP and the vertical signal line. A drive signal SEL is applied to the gate electrode of the selection transistor SEL. When the drive signal SEL becomes a high potential level, the selection transistor SEL becomes conductive, and the unit pixel becomes selected. As a result, the pixel signal output from the amplification transistor AMP is read out to the vertical signal line via the selection transistor.

[0054] The access transistor HV is composed of two gate electrodes. One gate electrode (OFG_H) is connected to the horizontal drive unit 14, and the other gate electrode (OHG_V) is connected to the vertical drive unit 12. OFG_H is an example of a first gate, and OHG_V is an example of a second gate.

[0055] The drain of the access transistor HV is connected to the power supply voltage VDD, and the source is connected to the photodiode PD.

[0056] When the potentials of the two gate electrodes of the access transistor HV become High under the control of the control unit 15, the access transistor HV is turned on and the charge of the photodiode PD is initialized. To distinguish it from the first embodiment, the access transistor HV in this embodiment is also called a gate-gate type access transistor HV.

[0057] The combinations of conductivity types of the transistors are merely examples, and are not limited to these. For example, some of the transistors may be PMOS transistors.

[0058] A vertical drive signal and a horizontal drive signal are supplied to the access transistor HV at irregular timing from the vertical drive unit 12 and the horizontal drive unit 14 under the control of the control unit 15. The irregular timing may be, for example, random timing.

[0059] FIG. 4 is a diagram illustrating the relationship between the state of the access transistor HV and the charge accumulation time in each pixel in the first embodiment.

[0060] 4A shows the relationship between the gate electrodes of the access transistor HV. The H-direction column in the figure shows the potential of OFG_H, and the V-direction column shows the potential of OHG_V. In this example, the access transistor HV is turned on when the potentials of OHG_H and OHG_V are both high, and current flows to the source side. Furthermore, the access transistor HV is turned off when the potential of at least one of OHG_H and OHG_V is low.

[0061] That is, when the access transistor HV is turned on, the charge in the photodiode PD is initialized, and charge accumulation starts from that timing.

[0062] FIG. 4B is a diagram illustrating the charge accumulation state of each photodiode PD in response to the application state of the first OFG transmitted from the horizontal and vertical drive units during the accumulation period in the pixel array unit 20 according to the first embodiment.

[0063] In this diagram, a pixel 100 with 7 rows and 7 columns will be described as an example. H or L written vertically indicates the potential of the access transistor HV in the V direction, that is, the potential of OHG_V, and H or L written horizontally indicates the potential of the access transistor HV in the H direction, that is, the potential of OHG_H. As described above, the access transistor HV is turned on when the potentials in the H direction and the V direction are both High, and the charge of the photodiode PD is initialized. Since the photodiode PD starts to accumulate charge again from that timing, the charge accumulation time is relatively short compared to the other photodiodes PD. In this diagram, pixels with a relatively short charge accumulation time (short-accumulated pixels) are represented as "short," and pixels with a relatively long charge accumulation time (long-accumulated pixels) are represented as "long."

[0064] FIG. 4C is a diagram illustrating the charge accumulation state of each photodiode PD in response to the application state of the second OFG transmitted from the horizontal and vertical drive units during the accumulation period in the pixel array unit 20 according to the first embodiment.

[0065] This diagram shows the charge accumulation state when the access transistor HV is turned on randomly after FIG. 4B . When the access transistor HV is turned on at this timing, the corresponding photodiode PD is initialized and becomes a pixel (ultra-short accumulated pixel) with an even shorter charge accumulation time than a short accumulated pixel. In this diagram, the ultra-short accumulated pixel is represented as "ultra-short."

[0066] The number of times the access transistor HV is switched is not limited to 1 or 2. The number of times the access transistor HV is switched in one horizontal period can be any number, and the variation in the charge accumulation time of the photodiode PD can be increased depending on the number of times the access transistor HV is switched.

[0067] Long-term accumulation pixels are more likely to obtain distance measurement information of an object at a long distance or an object with low reflectivity than short-term accumulation pixels. On the other hand, short-term accumulation pixels are less likely to saturate the charge accumulation unit due to light reflected by an object at a short distance or an object with high reflectivity than long-term accumulation pixels. A high dynamic range can be realized by synthesizing pixel signals output from long-term accumulation pixels and short-term accumulation pixels by the signal processing unit 16. For example, by the control unit 15 switching the access transistor HV multiple times at random timing in one horizontal period, a difference occurs in the charge accumulation time accordingly, and the dynamic range can be expanded.

[0068] FIG. 5 is a time chart of shutter (SHT) and read (READ) in one vertical period of the pixel 100 in the first embodiment.

[0069] In this figure, a time chart is shown regarding the exposure times of a long-term accumulated pixel, a short-term accumulated pixel, and a long-end and short-end accumulated pixel. That is, FIG. 5 is a timing chart of each pixel shown in FIG. 4C . SHT_res is the timing when the reset transistor RST is turned on, and SHT_ofg is the timing when the access transistor HV is turned on. As shown in the figure, after all the pixels 100 start exposure at the timing of SHT_res, at the respective timings of SHT_ofg, the charge of the pixel whose access transistor HV is turned on is initialized. As shown in the figure, by two SHT_ofg, the accumulation time of the charge of the pixel 100 can be set to three types of patterns: a long-term accumulated pixel, a short-term accumulated pixel, and a long-end accumulated pixel.

[0070] FIG. 6 is a diagram illustrating the dynamic range of the solid-state imaging device 10 according to the first embodiment.

[0071] In this example, as shown in FIG. 4B , a dynamic range for the application state of the first OFG transmitted from the horizontal and vertical drive units during the accumulation period will be described. Furthermore, in this example, the floating diffusion FD2 is not used, and the charge is transferred from the photodiode PD only to the floating diffusion FD1. Because the access transistor HV that accumulates the charge is turned on once, each pixel 100 is divided into two types: a long-term accumulated pixel and a short-term accumulated pixel. In this example, when the access transistor HV is not turned on, the dynamic range is 44 dB, whereas when the access transistor HV is turned on once, the dynamic range is 76 dB.

[0072] FIG. 7 is a schematic cross-sectional view of the pixel 100 in the vicinity of the photodiode PD according to the first embodiment.

[0073] The N-type channel layer (indicated by the dashed-dotted line in the figure) connecting OFG_H and OFG_V of the access transistor HV has a lower concentration than the N-type diffusion region (indicated by the dashed-dotted line in the figure) connected to the power supply voltage VDD. Furthermore, the polysilicon electrodes constituting OFG_H and OFG_V are spaced, for example, at intervals of approximately 50 nm to 100 nm, which is closer than the distance between the gate electrodes of two normal transistors. When a voltage is applied to OFG_H and OFG_V, the N-type channel layer is depleted, generating an electric field. This electrically connects the power supply voltage VDD and the photodiode PD, allowing initialization. Furthermore, to suppress leakage current, the polysilicon electrodes may be P-POLY.

[0074] FIG. 8 is a potential diagram of the pixel 100 in the first embodiment.

[0075] 8 indicates the positional relationship from the floating diffusion FD1 to the power supply voltage VDD, and the vertical direction indicates the magnitude of the potential. The voltage level decreases toward the top of the vertical direction, and increases toward the bottom. In FIG. 8, the flow of initializing the charge in the photodiode PD and restarting charge accumulation will be described.

[0076] 8 shows a state in which OFG_H is off and OFG_V is on. At this time, the charge accumulated in the photodiode PD flows to OFG_V and is stored in the N-type channel layer. Meanwhile, because OFG_H is off, the charge is blocked without moving to the power supply voltage VDD. In this state, when OFG_H is turned on, the charge in the photodiode PD moves to the power supply voltage VDD and is initialized, but when OFG_V is turned off, the charge stored in the N-type channel layer is returned to the photodiode PD.

[0077] FIG. 9 is an example of a circuit diagram of a pixel 100 according to a first modified example of the first embodiment.

[0078] In this modification, the pixel 100 includes a photodiode PD, a transfer transistor TGL, a selection transistor SEL, an amplification transistor AMP, a floating diffusion FD1, a floating diffusion FD2, a reset transistor RST, an access transistor HV, and a connection transistor FDG.

[0079] In this modification, pixel 100 reads out the charge stored in photodiode PD to floating diffusion FD1 by turning on transfer transistor TGL under the control of control unit 15, and also turns on connection transistor FDG to expand the capacity of floating diffusion FD2 and floating diffusion FD1, thereby enabling all of the charge stored in PD to be stored. In other words, by switching the conversion efficiency, noise can be reduced when the conversion efficiency is high, while all of the charge in the photodiode can be read out when the conversion efficiency is low.

[0080] FIG. 10 is a diagram illustrating the dynamic range of the solid-state imaging device 10 according to the first modified example of the first embodiment.

[0081] In this example, a dynamic range will be described for the application state of the first OFG transmitted from the horizontal and vertical drive units during the accumulation period as shown in FIG. 4B. Because the access transistor HV is turned on once, each pixel 100 is divided into two types: a long-term accumulated pixel and a short-term accumulated pixel.

[0082] During exposure of the photodiode PD, part of the charge accumulated in the photodiode PD is stored in the floating diffusions FD1 and FD2.

[0083] In this example, as shown by the solid line, when no charge is stored in the floating diffusions FD1 and FD2, the dynamic range is 76 dB, whereas, as shown by the dashed line, when charge is stored in the floating diffusions FD1 and FD2, the dynamic range is 105 dB.

[0084] FIG. 11 is an example of a circuit diagram of a pixel 100 according to a second modified example of the first embodiment.

[0085] In this modification, the pixel 100 includes a photodiode PD, a transfer transistor TGL, a selection transistor SEL, an amplification transistor AMP, a floating diffusion FD1, a floating diffusion FD2, a reset transistor RST, an access transistor HV, a connection transistor FDG, and a storage capacitor Cs.

[0086] The storage capacitor Cs is a lateral overflow integration capacitor (LOFIC), and is, for example, a capacitor having a metal-insulator-metal (MIM) structure. During charge storage in the photodiode PD and the floating diffusions FD1 and FD2, the overflowing charge can be stored in the storage capacitor Cs. The charge stored in the photodiode PD can also be initialized by turning on the access transistor HV, as described above.

[0087] FIG. 12 is a diagram showing the relationship between the amount of charge stored in the photodiode PD and the floating diffusion FD2 in the second modified example of the first embodiment.

[0088] 12, the horizontal axis represents time, and the vertical axis represents the amount of charge stored in the photodiode PD or the floating diffusion FD2. In this diagram, the charge storage period in the floating diffusion FD2 is shown in light gray, and the operation of the photodiode PD when discharging charge is shown in dark gray.

[0089] First, from time T1 to time T2, charge is stored in the photodiode PD under the control of the control unit 15. At this point, charge is stored in the photodiode PD until it is saturated. At time T2, the charge in the photodiode PD is saturated, and charge is stored in the floating diffusion FD2 toward time T3.

[0090] At time T3, the access transistor HV is turned on, and the charge accumulated in the photodiode PD is initialized. At this time, the charge accumulated in the photodiode PD may be stored in the storage capacitor Cs. Between times T3 and T4, the charge is again stored in the photodiode PD until it is saturated.

[0091] From time T4 to T5, similar to T2 to T3, charge is stored in the floating diffusion FD2. Also, from time T5 to T6, similar to T3 to T4, the charge stored in the photodiode PD is initialized and charge is stored again. From time T6 to T7, similar to T2 to T3, charge is stored in the floating diffusion FD2. After the charge in the floating diffusion FD2 is saturated, charge may be stored in the storage capacitor Cs.

[0092] FIG. 13 is an example of a circuit diagram of a pixel 100 according to a third modified example of the first embodiment.

[0093] In this modification, the pixel 100 includes a photodiode PD, a drain transistor OFG, a transfer transistor TGL, a selection transistor SEL, an amplification transistor AMP, a floating diffusion FD, a reset transistor RST, and an access transistor HV. The floating diffusion FD is an example of a first floating diffusion.

[0094] The access transistor HV in this modification is an NMOS transistor having a gate electrode connected to the vertical drive unit 12, a drain electrode connected to the horizontal drive unit 14, and a source electrode connected to the gate electrode of the discharge transistor OFG. The access transistor HV supplies a drive signal OFG to the gate electrode of the discharge transistor OFG in response to a vertical drive signal supplied from the vertical drive unit 12 and a horizontal drive signal supplied from the horizontal drive unit 14. The amount of charge stored in the pixel 100 is controlled by the access transistor HV being turned on randomly. The access transistor HV in this embodiment is also referred to as a drain-gate type access transistor HV to distinguish it from other embodiments. The drive signal output when the access transistor HV is turned on is an example of a third drive signal.

[0095] The drain transistor OFG is an NMOS transistor having a gate electrode connected to the source electrode of the access transistor HV, a source electrode connected to the power supply voltage VDD, and a drain electrode connected to the anode side of the photodiode PD, and initializes the charge of the photodiode PD in response to a drive signal OFG applied to the gate electrode.

[0096] FIG. 14 is a schematic cross-sectional view of the vicinity of the photodiode PD in the third modified example of the first embodiment.

[0097] 14 shows a simplified view of the semiconductor substrate 130 and wiring region 140 that constitute the pixel 100. In addition, in the same figure, an N-type semiconductor substrate 130 surrounded by a P-type diffusion region is shown. － In addition to the photodiode PD, the gate electrode of the transfer transistor TGL, and the gate electrode of the emission transistor OFG, which are constituted by the diffusion region, the source electrode, the drain electrode, and the gate electrode of the access transistor HV, the corresponding N ＋ The gate electrode of the access transistor HV is denoted as OFG_V, and the drain electrode of the access transistor HV is denoted as OFG_V.

[0098] The photodiode PD is －In the photodiode PD, electrons among the charges generated by photoelectric conversion are transferred to the N-type diffusion region. － It is stored in the diffusion region and transferred by the transfer transistor TGL.

[0099] The wiring region 140 includes wiring for transmitting signals to elements and an insulating layer for insulating the wiring. The insulating layer is made of silicon oxide (SiO ２ ) or other insulating material.

[0100] When the access transistor HV is off, there is a concern that the floating state of the discharge transistor OFG may cause leakage current from the gate electrode of the discharge transistor OFG. Therefore, in this embodiment, a deep N-well region made of a high-concentration N-type semiconductor is provided to isolate the potential of the access transistor HV from other regions in the pixel, in this example, the P-type semiconductor substrate 130. This makes it possible to suppress leakage current associated with a PN forward bias. Furthermore, a negative bias is applied to the drain electrode of the access transistor HV.

[0101] With this configuration, in the access transistor HV, the P-type semiconductor substrate 130 (0 V potential) and the N ＋ This makes it possible to suppress floating of charges due to leakage current caused by a PN forward bias of the diffusion region (negative potential).

[0102] In addition, in order to further suppress the escape of charges due to leakage current, the control unit 15 may control the drain electrode of the access transistor HV to be charged with a negative potential every horizontal period.

[0103] FIG. 15 is another schematic cross-sectional view of the vicinity of the photodiode PD in the third modified example of the first embodiment.

[0104] 15, a deep N-well region is not provided, and instead the gate electrode of the discharge transistor OFG is formed of P-POLY. By using the work function difference to perform pinning, it is possible to prevent leakage current from the gate electrode of the discharge transistor OFG due to the floating of the discharge transistor OFG when the access transistor HV is off.

[0105] In addition to forming the gate electrode of the drain transistor OFG using P-POLY, the drain electrode of the access transistor HV may be charged to GND every horizontal period under the control of the control unit 15. Also, a positive boost charge pump may be used to boost the voltage of the gate electrode of the drain transistor OFG to GND or higher, which further prevents leakage current.

[0106] According to the present embodiment and each of the modified examples, the solid-state imaging device 10 includes an access transistor HV that is turned on at irregular timing in the pixel array unit 20. By turning on the access transistor HV under the control of the control unit 15, the solid-state imaging device 10 can set a plurality of exposure time patterns for each pixel without setting each row or column to the same state, thereby achieving a high dynamic range.

[0107] Furthermore, according to this embodiment and each of the modifications, the solid-state imaging device 10 randomly turns on the access transistor HV under the control of the control unit 15, and therefore can achieve a high dynamic range with simple control.

[0108] Furthermore, according to this embodiment and the first and second modifications, the access transistor HV is configured with two gate electrodes, and the N-type channel layer constituting the transistor has a lower concentration than the N-type diffusion region connected to the power supply voltage VDD. This eliminates the need to provide a deep N-well region as in the first embodiment, and allows the circuit area to be reduced.

[0109] Furthermore, according to this embodiment and each of the variations, the solid-state imaging device 10 is provided with an access transistor HV in each pixel 100, and by turning on these transistors at irregular timings, the charge accumulation status can be changed over time and space.

[0110] Second Embodiment FIG. 16 is an example of a circuit diagram of a pixel 100 according to a second embodiment.

[0111] In this embodiment, the pixel 100 includes a photodiode PD, a selection transistor SEL, an amplification transistor AMP, a connection transistor FDG, a reset transistor RST, a floating diffusion FD1, a floating diffusion FD2, and an access transistor HV. The configuration of the solid-state imaging device 10 is the same as that described above, and therefore description thereof will be omitted.

[0112] In this embodiment, the drain electrode of the access transistor HV is connected to the anode side of the photodiode PD, and the drain electrode is connected to the floating diffusion FD1, the source electrode of the connection transistor FDG, and the gate electrode of the amplification transistor AMP. The pixel 100 may be configured without including the FD2 or the connection transistor FDG.

[0113] When the access transistor HV, the connection transistor FDG, and the reset transistor RST are turned on, the charge accumulated in the photodiode PD is initialized. The access transistor HV has a role similar to that of the transfer transistor TGL. Furthermore, for example, when only the reset transistor RST is turned on, the pixel 100 can initialize only the charge accumulated in the floating diffusion FD2. In this way, by controlling each transistor, the control unit 15 can initialize all the charges in the photodiode PD and the floating diffusions FD1 and FD2, or can selectively initialize only the floating diffusions FD1 and FD2.

[0114] FIG. 17 is an example of a circuit diagram of a pixel 100 according to a first modified example of the second embodiment.

[0115] In this modification, the pixel 100 includes a photodiode PD, a transfer transistor TGL, a selection transistor SEL, an amplification transistor AMP, a floating diffusion FD1, a floating diffusion FD2, an access transistor HV, and a connection transistor FDG. In this configuration, the pixel 100 has the access transistor HV instead of the reset transistor RST.

[0116] In this modification, the drain electrode of the access transistor HV is connected to the floating diffusion FD2, and the source electrode of the access transistor HV is connected to the power supply voltage VDD. The access transistor HV is turned on when the potentials of RST_H and RST_V are both High. When the access transistor HV is turned on, the charge of the floating diffusion FD2 is initialized. Furthermore, when the connection transistor FDG is turned on in addition to the access transistor HV being turned on, the charge of the floating diffusion FD1 is also initialized.

[0117] FIG. 18 is a diagram showing the relationship between the amount of charge stored in the photodiode PD and the floating diffusion FD2 in the first modified example of the second embodiment.

[0118] 18, the horizontal axis represents time, and the vertical axis represents the amount of charge stored in the photodiode PD or the floating diffusion FD2. In this diagram, the charge storage period in the floating diffusion FD2 is shown in light gray, and the operation of the photodiode PD when discharging the charge is shown in dark gray.

[0119] First, at time T1', in the pixel array unit 20, charge is stored in the photodiode PD under the control of the control unit 15. At this point, charge is stored in the photodiode PD until it is saturated. At time T2', the charge in the photodiode PD is saturated, and charge is stored in the floating diffusion FD2 toward time T3'.

[0120] At time T3', the access transistor HV is turned on, and the charge stored in the floating diffusion FD2 is initialized. Between times T3' and T4', charge is again stored in the floating diffusion FD2 until it is saturated.

[0121] At time T4', the access transistor HV is turned on, and the charge stored in the floating diffusion FD2 is initialized. From time T4' to T5', charge is stored in the floating diffusion FD2, similar to T3' to T4'.

[0122] FIG. 19 is an example of a circuit diagram of a pixel 100 according to a second modification of the second embodiment.

[0123] In this modification, the pixel 100 includes a photodiode PD, a transfer transistor TGL, a selection transistor SEL, an amplification transistor AMP, a floating diffusion FD1, a floating diffusion FD2, a reset transistor RST, and an access transistor HV. In this configuration, the pixel 100 has the access transistor HV arranged in place of the connection transistor FDG.

[0124] In this modification, the drain electrode of the access transistor HV is connected to the floating diffusion FD1, and the source electrode of the access transistor HV is connected to the floating diffusion FD2. The access transistor HV is turned on when the potentials of FDG_H and FDG_V are both high. When the access transistor HV is turned on, the floating diffusions FD1 and FD2 are connected, and the capacitance value of the entire floating diffusion increases. In other words, the access transistor HV has the role of changing the conversion capacitance from charge to voltage.

[0125] As the capacitance value of the conversion capacitance in the pixel 100 increases, the efficiency of conversion from charge to voltage decreases. Furthermore, when the access transistor HV is turned off, the floating diffusions FD1 and FD2 are disconnected, and the capacitance value decreases. As the capacitance value of the conversion capacitance in the pixel 100 decreases, the efficiency of conversion from charge to voltage increases. In this way, during the charge readout period, each pixel 100 can switch the conversion efficiency by turning the access transistor HV on or off.

[0126] When the conversion efficiency is switched, during the readout period, either a pixel signal with a low conversion gain (LCG) or a pixel signal with a high conversion gain (HCG) is output from each pixel 100. By combining these pixel signals in the signal processing unit 16, a high dynamic range can be achieved.

[0127] According to the present embodiment and each of the modified examples, the solid-state imaging device 10 includes an access transistor HV that is turned on at irregular timing, instead of the transfer transistor TGL. By turning on the access transistor HV under the control of the control unit 15, it is possible to set a plurality of patterns of exposure time for each pixel, and also to initialize the charge of the photodiode PD from the connection transistor FDG and reset transistor RST side.

[0128] Third Embodiment FIG. 20 shows an example of the configuration of an imaging device 2 according to a third embodiment.

[0129] In this embodiment, the imaging device 2 includes an imaging unit 300, a signal processing unit 310, and a recognition unit 320. Hereinafter, the magnitude of the signal output from the pixel 100 is also referred to as sensitivity. That is, each pixel 100 includes an access transistor HV, and each pixel 100 is initialized at irregular timing, so that the pixel array unit 20 includes a plurality of pixels 100 with different sensitivities.

[0130] The imaging unit 300 is, for example, a solid-state imaging device 10. The imaging unit 300 includes the pixels 100 described in the first and second embodiments and their respective modifications (hereinafter, a sensor in which the exposure time can be changed for each pixel 100 and in which the sensitivity varies from pixel to pixel is also referred to as a sensitivity variation expansion sensor), and outputs an image captured by the sensitivity variation expansion sensor as a sensor output. The image output from the imaging unit 300 is supplied to a signal processing unit 310.

[0131] The signal processing unit 310 includes a sensitivity variation correction unit and a saturation correction unit. The sensitivity variation correction unit corrects sensitivity variations in the captured image based on sensitivity variation correction data. The saturation correction unit performs saturation correction on the captured image, for which sensitivity variations have been corrected by the sensitivity variation correction unit, based on saturated pixel information. The captured image, for which sensitivity variation correction and saturation correction have been performed by the signal processing unit 310, is supplied to the recognition unit 320. These processes may be performed by the signal processing unit 16 in the solid-state imaging device 10 described above, or the signal processing unit 310 may be provided as a separate block.

[0132] FIG. 21 is a schematic diagram for explaining the operation of the imaging device 2 in the third embodiment.

[0133] The imaging device 2 according to the present disclosure captures an image using a sensitivity variation expanding sensor and acquires a captured image as a sensor output of the sensitivity variation expanding sensor (step S1). The captured image contains noise due to sensitivity variations on a pixel-by-pixel basis, as schematically shown in image 4a. Image 4a is, for example, an image that is extremely difficult to visually recognize.

[0134] The imaging device 2 corrects the sensitivity variation of the captured image (image 4a) acquired by the sensitivity variation expanding sensor using sensitivity variation correction data, which is correction data for correcting sensitivity variation on a pixel-by-pixel basis in the sensitivity variation expanding sensor (step S2). As schematically shown in image 4b, the image corrected using the sensitivity variation correction data has reduced noise compared to the uncorrected image 4a.

[0135] The sensitivity variation correction data may be obtained by, for example, calibrating the sensitivity variation expanding sensor. However, the sensitivity variation correction data may also be obtained by estimating (online estimating) in accordance with the operation of the sensitivity variation expanding sensor.

[0136] Image 4b ​​corrected in step S2 may contain saturated pixels whose pixel values ​​are saturated. Therefore, the imaging device 2 performs saturation correction on image 4b ​​corrected in step S2 using saturated pixel information that indicates saturated pixel regions caused by pixels that become saturated when exposed to a predetermined amount of light (step S3). An image corrected using the saturated pixel information is an image in which noise is almost completely eliminated, as schematically shown in image 4c. Unlike images 4a and 4b described above, image 4c is an image that is easy to view.

[0137] The saturated pixel information may be obtained by, for example, threshold determination of the sensor output when a certain amount of light or more is irradiated onto the sensitivity variation expanding sensor. Alternatively, the saturated pixel information may be obtained by calibration related to saturation of the sensitivity variation expanding sensor.

[0138] The image capturing device 2 may perform a recognition process on the image 4c corrected in step S3 (step S4). The recognition process may be performed using a neural network to which a pre-trained model is applied. Image 4d schematically shows a rectangular bounding box 5 set for an object detected by the recognition process.

[0139] Note that the recognition process according to the present disclosure is not limited to a process using a neural network, and for example, other means may be applied to the recognition process according to the present disclosure as long as the recognition process is capable of learning and is executed using a trained model.

[0140] In this embodiment, a network for communication such as the Internet is also referred to as a communication network, and is distinguished from a neural network.

[0141] The image 4c obtained by performing the sensitivity variation correction process (step S2) and the saturation correction process (step S3) on the image captured by the sensitivity variation expansion sensor has an expanded dynamic range compared to an image captured by a general sensor. Therefore, by applying the imaging device 2, it is possible to obtain a more accurate recognition result.

[0142] FIG. 22 is a block diagram illustrating an example of a second system configuration according to the third embodiment.

[0143] An imaging system 2b shown in Fig. 22 includes an imaging device 2 including an imaging unit 300, a signal processing device 311, and a recognition processing device 321. The signal processing device 311 has functions equivalent to the signal processing unit 310 in the imaging device 2 described using Fig. 20. Similarly, the recognition processing device 321 has functions equivalent to the recognition unit 320 in the imaging device 2.

[0144] That is, in the first system configuration described above, the imaging unit 300, the signal processing unit 310, and the recognition unit 320 are configured to be included in one imaging device 2. In contrast to this, in the second system configuration, the imaging device 2, the signal processing device 311, and the recognition processing device 321 are configured as devices independent of each other.

[0145] In the second system configuration, either wired communication or wireless communication may be used as the communication method between the imaging device 2 and the signal processing device 311, and between the signal processing device 311 and the recognition processing device 321. Furthermore, one or both of the signal processing device 311 and the recognition processing device 321 may be connected by communication via a communication network, or may be configured on a cloud network.

[0146] FIG. 23 is a block diagram illustrating an example of a third system configuration according to the third embodiment.

[0147] An imaging system 2 b shown in FIG. 23 includes an imaging device 2 including an imaging section 300 , and an information processing device 220 including a signal processing section 310 and a recognition section 320 .

[0148] The information processing device 220 is, for example, a computer, and the signal processing unit 310 and the recognition unit 320 are configured, for example, by a predetermined program running on the computer. However, without being limited to this, one or both of the signal processing unit 310 and the recognition unit 320 may be configured by hardware circuits that operate in cooperation with each other.

[0149] In the third system configuration, either wired communication or wireless communication may be used as the communication method between the imaging device 2 and the information processing device 220. The information processing device 220 may be a server device connected to the imaging device 10b via a communication network, or may be configured on a cloud network.

[0150] FIG. 24 is a block diagram showing an example of the hardware configuration of an information processing device 220 that can be applied to the third system configuration.

[0151] In Figure 24, the information processing device 220 includes a CPU 2000, a ROM (Read Only Memory) 2001, a RAM (Random Access Memory) 2002, a display control unit 2003, a storage device 2004, an input device 2005, a data I / F 2006, a communication I / F 2007, and a camera I / F 2008, each of which is communicatively connected to each other via a bus 2020.

[0152] The storage device 2004 is a non-volatile storage medium such as a flash memory or a hard disk drive. The CPU 2000 controls the overall operation of the information processing device 220 in accordance with programs stored in the storage device 2004 or the ROM 2001, using the RAM 2002 as a work memory.

[0153] The display control unit 2003 generates a display signal that can be displayed by the display device 2010, based on display control information generated by the CPU 2000 in accordance with a program. The display control unit 2003 supplies the generated display signal to the display device 2010. The display device 2010 displays a screen corresponding to the display signal supplied from the display control unit 2003.

[0154] The input device 2005 is a device that accepts input operations by the user, and may be, for example, a pointing device such as a mouse, a keyboard, or a touchpad. The input device 2005 outputs a control signal in response to the input operation by the user.

[0155] The data I / F 2006 inputs and outputs data to and from external devices via wired or wireless communication. The communication I / F 2007 performs communication via a communication network. The camera I / F 2008 is an interface for transmitting and receiving control commands and captured images to and from the imaging device 10b. The data I / F 2006 can also be used as the camera I / F 2008.

[0156] In the information processing device 220, the CPU 2000 executes the information processing program according to the present disclosure, thereby configuring the above-mentioned signal processing unit 310 and recognition unit 320, for example, as modules on the main storage area of ​​the RAM 2002.

[0157] The information processing program can be acquired from an external source via a communication network such as the Internet through communication via the communication I / F 2007, and installed on the information processing device 220. Alternatively, the information processing program may be provided from an external source via the data I / F 2006. Furthermore, the information processing program may be provided by being stored in a removable storage medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), or a USB (Universal Serial Bus) memory.

[0158] The configuration shown in FIG. 24 can also be applied to one or both of the signal processing device 311 and the recognition processing device 321 in the second system configuration described above.

[0159] The following description will be given by taking the imaging device 2 including the imaging unit 300, signal processing unit 310, and recognition unit 320 shown in FIG.

[0160] The signal processing unit 310 corrects sensitivity variations in the captured image to generate a sensitivity variation corrected image. The signal processing unit 310 also performs saturation correction on the sensitivity variation corrected image to generate a saturated region corrected image. These corrections are performed using known techniques.

[0161] FIG. 25 is a functional block diagram illustrating an example of the functions of the imaging device 2 according to the third embodiment.

[0162] 25, the imaging device 2 includes a sensitivity variation expansion sensor 400, a sensitivity variation correction unit 401, a saturated region correction unit 402, and a saturated region estimation unit 403. The imaging system 2b may further include a recognition unit 430.

[0163] Of these, the sensitivity variation correction unit 401, the saturated region correction unit 402, and the saturated region estimation unit 403 are functions included in the above-mentioned signal processing unit 310 or the signal processing device 311. Furthermore, the recognition unit 430 corresponds to the above-mentioned recognition unit 320 or the recognition processing device 321.

[0164] The following description will be given by taking the imaging device 2 including the imaging unit 300, signal processing unit 310, and recognition unit 320 shown in FIG.

[0165] The sensitivity variation expansion sensor 400 supplies a sensor output (captured image) obtained by capturing an image to a sensitivity variation correction unit 401 and a saturated region estimation unit 403 as a sensitivity variation image.

[0166] The sensitivity variation correction unit 401 performs sensitivity variation correction processing on the sensitivity variation image supplied from the sensitivity variation expansion sensor 400 by using a variation correction map 410. The variation correction map 410 is created, for example, in advance based on the sensitivity of each pixel 100 in the entire region of the pixel array unit 20, or at least each pixel 100 in the effective pixel region, and is stored in a storage medium such as a memory included in the imaging device 2. The sensitivity variation correction unit 401 supplies the sensitivity variation corrected image, in which the sensitivity variation of the sensitivity variation image has been corrected, to the saturated region correction unit 402.

[0167] The saturated region estimation unit 403 estimates saturated regions in the pixel array unit 20 based on the sensitivity variation image supplied from the sensitivity variation spread sensor 400. For example, the imaging device 2 creates in advance a saturation threshold map 420 that indicates regions in the sensitivity variation spread sensor 400 that become saturated due to a predetermined amount of light or more. The saturated region estimation unit 403 references the saturation threshold map 420 to estimate saturated pixels in the sensitivity variation image supplied from the sensitivity variation spread sensor 400, and generates a saturated region map based on the estimation result. The saturated region estimation unit 403 passes the generated saturated region map to the saturated region correction unit 402.

[0168] The saturated region correction unit 402 performs saturation correction on the sensitivity variation corrected image supplied from the sensitivity variation correction unit 401, using the saturated region map passed from the saturated region estimation unit 403. The saturated region correction unit 402 passes the saturated region corrected image, which is the sensitivity variation corrected image, to the recognition unit 430.

[0169] The recognition unit 430 performs recognition processing using a neural network to which a pre-trained learning model is applied on the saturated region corrected image supplied from the saturated region correction unit 402. The recognition unit 430 outputs the recognition result obtained by the recognition processing on the saturated region corrected image.

[0170] The learning data for training the recognition unit 430 may be acquired, for example, by constructing a database based on sensitivity variation images captured by the sensitivity variation magnification sensor 400 included in the imaging device 2 including the recognition unit 430. Without being limited to this, the learning data may be acquired using sensitivity variation images generated by simulating sensitivity variation based on images captured by a non-sensitivity variation sensor in which sensitivity variation is not expected.

[0171] In the above example, the recognition process is performed using the saturated region corrected image, but for example, the recognition unit 430 may perform the recognition process using a sensitivity variation image as input and output the recognition result. That is, a plane including all of the pixels 100 included in the pixel array unit 20 is input to the neural network.

[0172] For example, the recognition unit 430 may perform recognition processing etc. by combining planes in the initial stage based on input of planes of three channels such as RGB or four channels such as RGBW. The recognition unit 430 may train a neural network based on information in which each plane is combined in a predetermined manner, for example.

[0173] According to this embodiment, the imaging device 2 performs recognition processing using a learning model that receives as input images captured by pixels 100 with different sensitivities. Since the imaging device 2 performs recognition processing using images with a wide dynamic range, it is possible to obtain highly accurate recognition results.

[0174] Fourth Embodiment FIG. 26 shows an example of the configuration of an imaging device 2 according to a fourth embodiment.

[0175] The imaging device 2 in this embodiment includes an imaging section 300 and a blur removal section 440. The imaging section 300 is, for example, the solid-state imaging device 10. The imaging section 300 includes the pixels 100 described in the first and second embodiments and their respective modifications, and is capable of generating captured images with different exposure times for each pixel. The captured images output from the imaging section 300 are supplied to the blur removal section 440. The blur removal section 440 in this embodiment receives as input a plurality of captured images, such as moving images containing blur, and outputs a reconstructed image in which blur has been removed using a learning model.

[0176] FIG. 27 is a schematic diagram for explaining the operation of the imaging device 2 in the fourth embodiment.

[0177] The imaging unit 300 generates a plurality of captured images, such as moving images, with different time axes, and inputs these captured images to the blur removal unit 440. This diagram shows that one captured image is composed of pixels in the x direction and y direction, and further shows that a plurality of images are input to or output from the blur removal unit 440 along the time axis t.

[0178] The blur removal unit includes a learning model that receives a plurality of captured images as input and outputs a plurality of reconstructed images from which blur has been removed. Blur removal is achieved, for example, by a coded exposure method. The blur removal unit 440 may generate coded images for the input captured images using a sampling function, and then generate reconstructed images based on the coded images.

[0179] For example, the learning model uses a video as input and learns the features and structure of the video, such as edges, using a K-SVD algorithm. If the application scene of the imaging device 2 is specified (for example, detecting moving objects on a crosswalk), the learning model may perform learning using a specific type of video corresponding to the application scene.

[0180] According to this embodiment, the imaging device 2 outputs a plurality of reconstructed images from which blur has been removed, using a learning model that receives as input a plurality of captured images by the pixels 100 with different exposure times. By using the pixels 100 with different exposure times, a high dynamic range can be achieved, and a highly accurate reconstructed image can be output.

[0181] <<Configuration Example of Vehicle Control System>> FIG. 28 is a block diagram showing a configuration example of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

[0182] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to driving assistance and automatic driving of the vehicle 1.

[0183] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, a driving assistance / autonomous driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0184] The vehicle control ECU 21, communication unit 22, map information storage unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, memory unit 28, cruise assist / autonomous driving control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are interconnected via a communication network 41 for mutual communication. The communication network 41 is configured, for example, by an in-vehicle communication network or bus conforming to a digital two-way communication standard such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), FlexRay (registered trademark), or Ethernet (registered trademark). Different communication networks 41 may be used depending on the type of data being transmitted. For example, a CAN may be used for data related to vehicle control, and an Ethernet may be used for large-volume data. In addition, each part of the vehicle control system 11 may be directly connected without going through the communication network 41, using wireless communication intended for communication over relatively short distances, such as near field communication (NFC) or Bluetooth (registered trademark).

[0185] In the following description, when each unit of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described that the vehicle control ECU 21 and the communication unit 22 communicate with each other.

[0186] The vehicle control ECU 21 is configured by various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The vehicle control ECU 21 controls the entire or part of the functions of the vehicle control system 11.

[0187] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 22 can communicate using a plurality of communication methods.

[0188] The following provides an overview of communication with the outside of the vehicle that can be performed by the communication unit 22. The communication unit 22 communicates with a server (hereinafter referred to as an external server) or the like on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the Internet, a cloud network, or a network specific to an operator. The communication method used by the communication unit 22 with the external network is not particularly limited as long as it is a wireless communication method that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0189] Furthermore, for example, the communication unit 22 can communicate with a terminal located near the vehicle using P2P (Peer to Peer) technology. The terminal located near the vehicle can be, for example, a terminal worn by a mobile object moving at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed location in a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 22 can also perform V2X communication. V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian.

[0190] The communication unit 22 can receive, for example, a program for updating software that controls the operation of the vehicle control system 11 from the outside (over the air). The communication unit 22 can also receive map information, traffic information, information about the surroundings of the vehicle 1, and the like from the outside. For example, the communication unit 22 can also transmit information about the vehicle 1 and information about the surroundings of the vehicle 1 to the outside. Information about the vehicle 1 that the communication unit 22 transmits to the outside includes, for example, data indicating the state of the vehicle 1 and the recognition result by the recognition unit 73. Furthermore, for example, the communication unit 22 performs communication corresponding to a vehicle emergency notification system such as e-call.

[0191] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System) (registered trademark)) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0192] The following provides an overview of communication with the vehicle interior that can be performed by the communication unit 22. The communication unit 22 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 22 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wireless communication, such as wireless LAN, Bluetooth, NFC, or Wireless USB (WUSB). The communication unit 22 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 22 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wired communication, such as Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI) (registered trademark), or Mobile High-Definition Link (MHL).

[0193] Here, the in-vehicle device refers to, for example, a device in the vehicle that is not connected to the communication network 41. Possible in-vehicle devices include, for example, a mobile device or wearable device carried by a passenger such as a driver, and an information device brought into the vehicle and temporarily installed therein.

[0194] The map information storage unit 23 stores one or both of a map acquired from an external source and a map created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.

[0195] Examples of high-precision maps include dynamic maps, point cloud maps, and vector maps. A dynamic map is a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. A point cloud map is a map made up of a point cloud (point group data). A vector map is a map that corresponds traffic information such as the positions of lanes and traffic lights to the point cloud map, and is adapted to an advanced driver assistance system (ADAS) or autonomous driving (AD).

[0196] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created in the vehicle 1 based on sensing results from the camera 51, radar 52, LiDAR 53, etc. as a map for matching with a local map described later, and stored in the map information storage unit 23. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred square meters relating to the planned route along which the vehicle 1 will travel is acquired from the external server or the like in order to reduce communication capacity.

[0197] The position information acquisition unit 24 receives GNSS (Global Navigation Satellite System) signals from GNSS satellites and acquires position information of the vehicle 1. The acquired position information is supplied to the driving assistance / autonomous driving control unit 29. Note that the method of the position information acquisition unit 24 is not limited to using GNSS signals, and it may acquire position information using a beacon, for example.

[0198] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0199] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. Without being limited to this, the external recognition sensor 25 may be configured to include one or more types of sensors selected from the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 is not particularly limited as long as the number is a number that can be realistically installed on the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0200] The imaging method of the camera 51 is not particularly limited. For example, cameras of various imaging methods capable of distance measurement, such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera, can be applied to the camera 51 as needed. However, the camera 51 may simply acquire an image without distance measurement.

[0201] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment for the vehicle 1. The environmental sensor is a sensor for detecting the environment such as weather, climate, brightness, etc., and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0202] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the location of sound sources.

[0203] The interior sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the interior sensor 26 are not particularly limited as long as they are of types and numbers that can be realistically installed in the vehicle 1.

[0204] For example, the interior sensor 26 may include one or more types of sensors selected from the group consisting of a camera, radar, a seating sensor, a steering wheel sensor, a microphone, and a biometric sensor. The camera included in the interior sensor 26 may be a camera using any of various imaging methods capable of measuring distances, such as a Time of Flight (ToF) camera, a stereo camera, a monocular camera, or an infrared camera. The camera included in the interior sensor 26 may also be a camera simply for acquiring captured images, regardless of distance measurement. The biometric sensor included in the interior sensor 26 may be provided, for example, on a seat, a steering wheel, or the like, and detect various types of biometric information of a passenger, such as a driver.

[0205] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the vehicle sensor 27 are not particularly limited as long as they are of types and numbers that can be realistically installed on the vehicle 1.

[0206] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these sensors. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining battery charge and temperature, and an impact sensor that detects external impacts.

[0207] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The storage unit 28 is used, for example, as an electrically erasable programmable read-only memory (EEPROM) and a random access memory (RAM). Examples of storage media that can be used include a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, and a magneto-optical storage device. The storage unit 28 stores various programs and data used by each component of the vehicle control system 11. For example, the storage unit 28 includes an event data recorder (EDR) and a data storage system for automated driving (DSSAD), and stores information about the vehicle 1 before and after an event such as an accident, and information acquired by the in-vehicle sensors 26.

[0208] The driving assistance / automated driving control unit 29 controls driving assistance and automatic driving of the vehicle 1. For example, the driving assistance / automated driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0209] The analysis unit 61 performs an analysis process of the vehicle 1 and the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0210] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25 and matches the local map with the high-precision map to estimate the self-position of the vehicle 1. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

[0211] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technique such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size and the occupancy status of objects is indicated on a grid-by-grid basis. The occupancy status of objects is indicated, for example, by the presence or absence of an object and its probability of existence. The local map is also used, for example, in the detection process and recognition process of the situation outside the vehicle 1 by the recognition unit 73.

[0212] The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27 .

[0213] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and association.

[0214] The recognition unit 73 executes a detection process for detecting the situation outside the vehicle 1 and a recognition process for recognizing the situation outside the vehicle 1 .

[0215] For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0216] Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0217] For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering to classify a point cloud based on sensor data from the radar 52, the LiDAR 53, or the like into clusters of points. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0218] For example, the recognition unit 73 performs tracking to follow the movement of clusters of point clouds classified by clustering, thereby detecting the movement of objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0219] For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. based on image data supplied from the camera 51. The recognition unit 73 may also recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0220] For example, the recognition unit 73 can perform a recognition process of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the result of estimation of the self-position by the self-position estimation unit 71, and the result of recognition of objects around the vehicle 1 by the recognition unit 73. Through this process, the recognition unit 73 can recognize the positions and states of traffic lights, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes that can be traveled, etc.

[0221] For example, the recognition unit 73 can perform a recognition process of the environment around the vehicle 1. The surrounding environment to be recognized by the recognition unit 73 may include weather, temperature, humidity, brightness, and road surface conditions.

[0222] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route tracking processing.

[0223] Global path planning is a process for planning a rough route from the start to the goal. This route planning also includes a process for generating a trajectory (local path planning) that allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1, taking into account the motion characteristics of the vehicle 1 on the planned route.

[0224] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route plan within a planned time. The behavior planning unit 62 can, for example, calculate a target speed and a target angular velocity of the vehicle 1 based on the results of this path following process.

[0225] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62 .

[0226] For example, the operation control unit 63 controls the steering control unit 81, the brake control unit 82, and the drive control unit 83 included in the vehicle control unit 32 described later to perform acceleration / deceleration control and direction control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs cooperative control with the aim of realizing ADAS functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, collision warning for the host vehicle, and lane departure warning for the host vehicle. For example, the operation control unit 63 performs cooperative control with the aim of automatic driving, which travels autonomously without driver operation.

[0227] The DMS 30 performs processes such as authenticating the driver and recognizing the driver's state based on sensor data from the in-vehicle sensors 26 and input data input to the HMI 31 (described later). Examples of the driver's state to be recognized include physical condition, level of alertness, level of concentration, level of fatigue, line of sight, level of intoxication, driving operation, and posture.

[0228] The DMS 30 may be configured to perform authentication processing for passengers other than the driver and recognition processing for the conditions of the passengers. Furthermore, for example, the DMS 30 may be configured to perform recognition processing for the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Possible conditions inside the vehicle to be recognized include, for example, temperature, humidity, brightness, and odor.

[0229] The HMI 31 inputs various data and instructions and presents various data to the driver and the like.

[0230] The following provides an overview of data input via the HMI 31. The HMI 31 includes input devices for a person to input data. The HMI 31 generates input signals based on data, instructions, and the like input via the input devices and supplies the signals to each component of the vehicle control system 11. The HMI 31 includes, as input devices, controls such as a touch panel, buttons, switches, and levers. The HMI 31 may also include input devices that allow information to be input by voice, gestures, or other means other than manual operation. Furthermore, the HMI 31 may use, as input devices, externally connected devices such as a remote control device using infrared or radio waves, or a mobile or wearable device compatible with the operation of the vehicle control system 11.

[0231] The presentation of data by the HMI 31 will be briefly described. The HMI 31 generates visual information, auditory information, and tactile information for the occupant or the outside of the vehicle. The HMI 31 also performs output control, controlling the output, output content, output timing, output method, etc. of each piece of generated information. The HMI 31 generates and outputs, as visual information, information indicated by images or lights, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image showing the situation around the vehicle 1. The HMI 31 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 31 also generates and outputs, as tactile information, information imparted to the occupant's sense of touch by, for example, force, vibration, movement, etc.

[0232] Examples of the output device to which the HMI 31 outputs visual information include a display device that presents visual information by displaying an image on its own or a projector device that presents visual information by projecting an image. The display device may be a device that displays visual information within the field of view of the occupant, such as a head-up display, a transmissive display, or a wearable device with an augmented reality (AR) function, in addition to a display device having a normal display. The HMI 31 may also use display devices included in a navigation system, an instrument panel, a camera monitoring system (CMS), an electronic mirror, a lamp, or the like provided in the vehicle 1 as output devices that output visual information.

[0233] As an output device for the HMI 31 to output auditory information, for example, an audio speaker, a headphone, or an earphone can be applied.

[0234] For example, a haptic element using haptic technology can be applied as an output device for outputting tactile information from the HMI 31. The haptic element is provided on a part of the vehicle 1 that an occupant comes into contact with, such as a steering wheel or a seat.

[0235] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0236] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0237] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an antilock brake system (ABS), a regenerative brake mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0238] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a drive ECU for controlling the drive system, and an actuator for driving the drive system.

[0239] The body system control unit 84 detects and controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a body system ECU that controls the body system systems, an actuator that drives the body system systems, etc.

[0240] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a light ECU that controls the lights, an actuator that drives the lights, and the like.

[0241] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and the like.

[0242] Fig. 29 is a diagram showing an example of a sensing area by the camera 51, radar 52, LiDAR 53, ultrasonic sensor 54, etc. of the external recognition sensor 25 in Fig. 28. Note that Fig. 28 schematically shows the vehicle 1 as seen from above, with the left end side being the front end (front) side of the vehicle 1 and the right end side being the rear end (rear) side of the vehicle 1.

[0243] Sensing area 101F and sensing area 101B show examples of sensing areas of the ultrasonic sensors 54. Sensing area 101F covers the periphery of the front end of the vehicle 1 with multiple ultrasonic sensors 54. Sensing area 101B covers the periphery of the rear end of the vehicle 1 with multiple ultrasonic sensors 54.

[0244] The sensing results in the sensing area 101F and the sensing area 101B are used, for example, for parking assistance for the vehicle 1.

[0245] Sensing area 102F to sensing area 102B show examples of sensing areas of a short-range or medium-range radar 52. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther behind the vehicle 1 than sensing area 101B. Sensing area 102L covers the periphery behind the left side of the vehicle 1. Sensing area 102R covers the periphery behind the right side of the vehicle 1.

[0246] The sensing results in sensing area 102F are used, for example, to detect vehicles, pedestrians, and the like that are present in front of the vehicle 1. The sensing results in sensing area 102B are used, for example, for a collision prevention function behind the vehicle 1. The sensing results in sensing area 102L and sensing area 102R are used, for example, to detect objects in blind spots on the sides of the vehicle 1.

[0247] Sensing areas 103F to 103B show examples of sensing areas sensed by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther in the rear of vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 1. Sensing area 103R covers the periphery of the right side of vehicle 1.

[0248] The sensing results in the sensing area 103F can be used for, for example, recognition of traffic lights and traffic signs, lane departure prevention assistance systems, and automatic headlight control systems. The sensing results in the sensing area 103B can be used for, for example, parking assistance and surround view systems. The sensing results in the sensing areas 103L and 103R can be used for, for example, surround view systems.

[0249] Sensing area 104 shows an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a position farther ahead of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

[0250] The sensing results in the sensing area 104 are used to detect objects such as surrounding vehicles, for example.

[0251] A sensing area 105 shows an example of the sensing area of ​​the long-range radar 52. The sensing area 105 covers a position further ahead of the vehicle 1 than the sensing area 104. On the other hand, the sensing area 105 has a narrower range in the left-right direction than the sensing area 104.

[0252] The sensing results in the sensing area 105 are used for, for example, adaptive cruise control (ACC), emergency braking, collision avoidance, and the like.

[0253] The sensing areas of the cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 included in the external recognition sensor 25 may have various configurations other than those shown in FIG. 28 . Specifically, the ultrasonic sensors 54 may also sense the sides of the vehicle 1, and the LiDAR 53 may sense the rear of the vehicle 1. The installation positions of the sensors are not limited to the above-described examples. The number of each sensor may be one or more.

[0254] The present disclosure has been described above by giving embodiments and their modifications, application examples, and applied examples. However, the present disclosure is not limited to the above-described embodiments, etc., and various modifications are possible. Note that the effects described in this specification are merely examples. The effects of the present disclosure are not limited to the effects described in this specification. The present disclosure may have effects other than those described in this specification.

[0255] Furthermore, for example, the present disclosure can be configured as follows.

[0256] (1) A solid-state imaging device comprising: a pixel array section in which a plurality of pixels are arranged in a two-dimensional array; a vertical drive section that supplies a first drive signal to the plurality of pixels in the vertical direction; a horizontal drive section that supplies a second drive signal to the plurality of pixels in the horizontal direction; and a control section that controls the horizontal drive section and the vertical drive section, wherein each of the plurality of pixels includes an access transistor that controls the amount of charge accumulated in the pixel based on the first drive signal and the second drive signal, and the control section controls the vertical drive section and the horizontal drive section to output the first drive signal and the second drive signal at irregular timing.

[0257] (2) The solid-state imaging device described in (1), wherein the access transistor includes a first gate that receives input of the first drive signal and a second gate that receives input of the second drive signal, and the concentration of an N-type channel layer connecting the first gate and the second gate is lower than the concentration of an N-type diffusion region connected to a power supply voltage.

[0258] (3) The solid-state imaging device according to (2), wherein each of the plurality of pixels further includes a first floating diffusion that accumulates charge transferred from a photodiode, and a second floating diffusion that accumulates charge that overflows from the first floating diffusion.

[0259] (4) The solid-state imaging device according to (3), wherein one of a source and a drain of the access transistor is connected to the power supply voltage, and the other of the source and the drain is connected to the photodiode.

[0260] (5) The solid-state imaging device according to (3), wherein one of a source and a drain of the access transistor is connected to the first floating diffusion, and the other of the source and the drain is connected to the photodiode.

[0261] (6) The solid-state imaging device according to (1), wherein the access transistor initializes the charge accumulated in the photodiode based on the first drive signal and the second drive signal.

[0262] (7) The solid-state imaging device according to (1), wherein each of the plurality of pixels further includes a drain transistor that connects a photodiode to a power supply voltage based on a third drive signal supplied from the access transistor.

[0263] (8) The solid-state imaging device according to (7), wherein the access transistor has a gate connected to the vertical drive unit, one of a source and a drain connected to the horizontal drive unit, and the other of the source and drain connected to the gate of the discharge transistor, and outputs the third drive signal.

[0264] (9) The solid-state imaging device according to (7), wherein each of the plurality of pixels has a well structure that separates the potential of the access transistor from a region within the pixel.

[0265] (10) The solid-state imaging device according to (7), wherein the gate of the drain transistor is formed of P-POLY.

[0266] (11) The solid-state imaging device according to (1), wherein the irregular timing is random timing.

[0267] (12) The solid-state imaging device according to (3), wherein each of the plurality of pixels further includes a storage capacitor that stores charge overflowing from the first floating diffusion or the second floating diffusion, and the storage capacitor has an MIM structure.

[0268] (13) The solid-state imaging device described in (3), wherein each of the plurality of pixels further includes a connection transistor connecting the first floating diffusion and the second floating diffusion, and wherein one of the source and drain of the access transistor is connected to the power supply voltage, and the other of the source and drain is connected to the second floating diffusion.

[0269] (14) The solid-state imaging device described in (3), wherein each of the plurality of pixels further includes a reset transistor that initializes the charge accumulated in the second floating diffusion, and the access transistor has one of a source and a drain connected to the first floating diffusion, and the other of the source and drain connected to the second floating diffusion.

[0270] (15) An electronic device having a solid-state imaging device, the electronic device comprising: a recognition unit including a learning model that receives an image as an input and outputs a recognition result, the solid-state imaging device comprising: a pixel array unit in which a plurality of pixels are arranged in a two-dimensional array; a vertical drive unit that supplies a first drive signal to the plurality of pixels in the vertical direction; a horizontal drive unit that supplies a second drive signal to the plurality of pixels in the horizontal direction; and a control unit that controls the horizontal drive unit and the vertical drive unit, each of the plurality of pixels including an access transistor that controls the amount of charge accumulated in the pixel based on the first drive signal and the second drive signal, and the control unit controls the vertical drive unit and the horizontal drive unit to output the first drive signal and the second drive signal at irregular timing.

[0271] (16) The electronic device described in (15), wherein the access transistor includes a first gate that receives an input of the first drive signal and a second gate that receives an input of the second drive signal, and an N-type channel layer connecting the first gate and the second gate has a lower concentration than an N-type diffusion region connected to a power supply voltage.

[0272] (17) The electronic device according to (16), wherein one of a source and a drain of the access transistor is connected to the power supply voltage, and the other of the source and the drain is connected to a photodiode.

[0273] (18) An electronic device having a solid-state imaging device, the electronic device comprising: a blur removal unit including a learning model that receives a plurality of captured images as input and outputs a plurality of reconstructed images from which blur has been removed; the solid-state imaging device comprising: a pixel array unit in which a plurality of pixels are arranged in a two-dimensional array; a vertical drive unit that supplies a first drive signal to the plurality of pixels in the vertical direction; a horizontal drive unit that supplies a second drive signal to the plurality of pixels in the horizontal direction; and a control unit that controls the horizontal drive unit and the vertical drive unit; each of the plurality of pixels including an access transistor that controls the amount of charge accumulated in the pixel based on the first drive signal and the second drive signal; and the control unit that controls the vertical drive unit and the horizontal drive unit to output the first drive signal and the second drive signal at irregular timing.

[0274] (19) The electronic device described in (18), wherein the access transistor includes a first gate that receives an input of the first drive signal and a second gate that receives an input of the second drive signal, and an N-type channel layer connecting the first gate and the second gate has a lower concentration than an N-type diffusion region connected to a power supply voltage.

[0275] (20) The electronic device according to (19), wherein one of a source and a drain of the access transistor is connected to the power supply voltage, and the other of the source and the drain is connected to a photodiode.

[0276] 1: Vehicle, 2: Imaging device, 4a: Image, 4b: Image, 4c: Image, 4d: Image, 5: Bounding box, 11: Vehicle control system, 10: Solid-state imaging device, 12: Vertical drive unit, 13: AD conversion unit, 14: Horizontal drive unit, 15: Control unit, 16: Signal processing unit, 17: Data storage unit, 18: Input / output unit, 20: Pixel array unit, 21: Vehicle control ECU, 22: Communication unit, 23: Map information storage unit, 24: Position information acquisition unit, 25: External recognition sensor, 26: In-vehicle sensor, 27: Vehicle sensor, 28: Memory unit, 29: Cruise assistance / autonomous driving control unit, 30: DMS, 31: HMI, 32: Vehicle control unit, 61: Analysis unit, 62: Action planning unit, 63: Operation control unit, 71: Self-position estimation unit, 72: Sensor fusion unit, 73: Recognition unit, 81: Steering control unit, 82: Brake control unit, 83: Drive control unit, 84: Body system control unit, 85: Light control unit, 86: Horn control unit, 100: Pixel, 130: Semiconductor substrate, 139: Signal line, 140: Wiring area, 149: Insulating layer, 209: Signal line, 210: Imaging lens, 10: Solid-state imaging device, 220: Information processing device, 230: Imaging control unit, 240: Recording unit, 300: Imaging unit, 310: Signal processing unit, 320: Recognition unit, 400: Sensitivity variation expansion sensor, 401: Sensitivity variation correction unit, 402: Saturation area correction unit, 403: Saturation area estimation unit, 410: Variation correction map, 420: Saturation threshold map, 430: Recognition unit, 440: Blur removal unit, 2000: CPU, 2001: ROM, 2002: RAM, 2003: Display control unit, 2004: Storage device, 2005: Input device, 2006: Data I / F, 2007: Communication I / F, 2008: Camera I / F, 2010: Display device, 2020: Bus

Claims

1. A solid-state imaging device comprising: a pixel array section in which a plurality of pixels are arranged in a two-dimensional array; a vertical drive section that supplies a first drive signal to the plurality of pixels in the vertical direction; a horizontal drive section that supplies a second drive signal to the plurality of pixels in the horizontal direction; and a control section that controls the horizontal drive section and the vertical drive section, wherein each of the plurality of pixels includes an access transistor that controls the amount of charge accumulated in the pixel based on the first drive signal and the second drive signal, and the control section controls the vertical drive section and the horizontal drive section to output the first drive signal and the second drive signal at irregular timing.

2. The solid-state imaging device of claim 1, wherein the access transistor includes a first gate that receives input of the first drive signal and a second gate that receives input of the second drive signal, and the concentration of an N-type channel layer connecting the first gate and the second gate is lower than the concentration of an N-type diffusion region connected to a power supply voltage.

3. The solid-state imaging device according to claim 2, wherein each of the plurality of pixels further includes a first floating diffusion that accumulates charge transferred from a photodiode, and a second floating diffusion that accumulates charge that overflows from the first floating diffusion.

4. The solid-state imaging device according to claim 3, wherein one of the source and drain of the access transistor is connected to the power supply voltage, and the other of the source and drain is connected to the photodiode.

5. The solid-state imaging device according to claim 3, wherein one of the source and drain of the access transistor is connected to the first floating diffusion, and the other of the source and drain is connected to the photodiode.

6. The solid-state imaging device according to claim 1, wherein the access transistor initializes the charge stored in the photodiode based on the first drive signal and the second drive signal.

7. The solid-state imaging device according to claim 1, wherein each of the plurality of pixels further includes a drain transistor that connects the photodiode to a power supply voltage based on a third drive signal supplied from the access transistor.

8. The solid-state imaging device according to claim 7, wherein the access transistor has a gate connected to the vertical drive section, one of a source and a drain connected to the horizontal drive section, and the other of a source and a drain connected to the gate of the discharge transistor, and outputs the third drive signal.

9. The solid-state imaging device according to claim 7, wherein each of the plurality of pixels has a well structure that separates the potential of the access transistor from a region within the pixel.

10. The solid-state imaging device according to claim 7, wherein the gate of said drain transistor is formed of P-POLY.

11. The solid-state imaging device according to claim 1, wherein the irregular timing is random timing.

12. The solid-state imaging device according to claim 3, wherein each of the plurality of pixels further includes a storage capacitor that stores charge that overflows from the first floating diffusion or the second floating diffusion, and the storage capacitor has an MIM (Metal Insulator Metal) structure.

13. A solid-state imaging device as described in claim 3, wherein each of the plurality of pixels further includes a connection transistor connecting the first floating diffusion and the second floating diffusion, and wherein one of the source and drain of the access transistor is connected to the power supply voltage, and the other of the source and drain is connected to the second floating diffusion.

14. A solid-state imaging device as described in claim 3, wherein each of the plurality of pixels further includes a reset transistor that initializes the charge accumulated in the second floating diffusion, and wherein the access transistor has one of its source and drain connected to the first floating diffusion and the other of its source and drain connected to the second floating diffusion.

15. An electronic device comprising a solid-state imaging device, the electronic device comprising a recognition unit including a learning model that receives an image as input and outputs a recognition result, the solid-state imaging device comprising: a pixel array unit in which a plurality of pixels are arranged in a two-dimensional array; a vertical drive unit that supplies a first drive signal to the plurality of pixels in the vertical direction; a horizontal drive unit that supplies a second drive signal to the plurality of pixels in the horizontal direction; and a control unit that controls the horizontal drive unit and the vertical drive unit, each of the plurality of pixels including an access transistor that controls the amount of charge accumulated in the pixel based on the first drive signal and the second drive signal, and the control unit controls the vertical drive unit and the horizontal drive unit to output the first drive signal and the second drive signal at irregular timing.

16. The electronic device according to claim 15, wherein the access transistor includes a first gate that receives an input of the first drive signal and a second gate that receives an input of the second drive signal, and wherein an impurity concentration of an N-type channel layer connecting the first gate and the second gate is lower than an impurity concentration of an N-type diffusion region connected to a power supply voltage.

17. The electronic device according to claim 16, wherein one of the source and drain of the access transistor is connected to the power supply voltage, and the other of the source and drain is connected to a photodiode.

18. An electronic device having a solid-state imaging device, comprising a blur removal unit including a learning model that receives a plurality of captured images as input and outputs a plurality of reconstructed images from which blur has been removed, wherein the solid-state imaging device comprises: a pixel array unit in which a plurality of pixels are arranged in a two-dimensional array; a vertical drive unit that supplies a first drive signal to the plurality of pixels in the vertical direction; a horizontal drive unit that supplies a second drive signal to the plurality of pixels in the horizontal direction; and a control unit that controls the horizontal drive unit and the vertical drive unit, wherein each of the plurality of pixels includes an access transistor that controls the amount of charge accumulated in the pixel based on the first drive signal and the second drive signal, and the control unit controls the vertical drive unit and the horizontal drive unit to output the first drive signal and the second drive signal at irregular timing.

19. The electronic device according to claim 18, wherein the access transistor includes a first gate that receives input of the first drive signal and a second gate that receives input of the second drive signal, and wherein an N-type channel layer connecting the first gate and the second gate has a lower concentration than an N-type diffusion region connected to a power supply voltage.

20. The electronic device according to claim 19, wherein one of the source and drain of the access transistor is connected to the power supply voltage, and the other of the source and drain is connected to a photodiode.

Images