Solid-state imaging device

Abstract

[Problem] To acquire brightness information and brightness change information at the same timing and also with high resolution. [Solution] This solid-state imaging device is provided with a plurality of pixels and a control circuit. Each of the plurality of pixels includes: a photoelectric conversion element for applying photoelectric conversion to incident light to generate an electric signal; the first pixel circuit is used for converting the electric signal into first information; and a second pixel circuit for converting the electrical signal into second information. The control circuit performs control to connect the photoelectric conversion element to any one of the first pixel circuit and the second pixel circuit for each of the plurality of pixels.

Description

Technical Field

[0001] The present disclosure relates to a solid-state imaging device. Background Art

[0002] Image sensors are generally intended to obtain brightness information. Nowadays, sensors that do not obtain brightness information but obtain brightness change information according to the purpose are widely studied. Image sensors for obtaining these two types of information are being studied. In these image sensors, for example, two types of information are obtained by switching between a drive for obtaining brightness information and a drive for obtaining brightness change information, or by separately arranging a pixel area for obtaining brightness information and a pixel area for obtaining brightness change information.

[0003] However, in the switching system, there is a problem that the brightness information and the brightness change information cannot be obtained at the same time, and in the area segmentation system, there is a problem that the resolution of the brightness information and the brightness change information is always low. If the brightness information and the brightness change information cannot be obtained at the same time, for example, in the case of capturing an image of an object moving at a high speed, a time lag occurs between the brightness information and the brightness change information, and it is difficult to generate a high-quality image based on the obtained information or to achieve accurate object recognition, etc. In addition, in the case of reduced resolution, image quality and recognition rate may be similarly reduced.

[0004] Reference List

[0005] Patent Literature

[0006] Patent document 1: WO 2019 / 146527 A

[0007] Patent Document 2: Japanese Patent Application Publication No. 2021-129265 Summary of the invention

[0008] Problems to be solved by the present invention

[0009] Therefore, the present disclosure provides a solid-state imaging device that obtains luminance information and luminance change information at the same timing and with high resolution.

[0010] Solution to the problem

[0011] According to an embodiment, a solid-state imaging device includes a plurality of pixels and a control circuit. Each of the plurality of pixels includes: a photoelectric conversion element that performs photoelectric conversion on incident light to generate an electrical signal; a first pixel circuit that converts the electrical signal into first information; and a second pixel circuit that converts the electrical signal into second information. The control circuit controls each of the plurality of pixels so that the photoelectric conversion element is connected to the first pixel circuit or the second pixel circuit.

[0012] The control circuit may control whether the first pixel circuit or the second pixel circuit is to convert the electrical signal based on the arrangement of each pixel.

[0013] The control circuit may control switching between an output timing using the first pixel circuit and an output timing using the second pixel circuit for each of the plurality of pixels.

[0014] A pixel array in which a plurality of pixels are arranged may be included. The plurality of pixels may be arranged in a two-dimensional array along rows in the pixel array and columns extending in a direction intersecting the direction of the rows.

[0015] The control circuit may perform control for switching between pixels that perform output using the first pixel circuit and pixels that perform output using the second pixel circuit for each region in the pixel array.

[0016] The photoelectric conversion element may be an element that obtains brightness information. The first pixel circuit may convert the electrical signal into grayscale information, and the second pixel circuit may convert the electrical signal into event detection information.

[0017] The photoelectric conversion element may be a single photon avalanche diode (SPAD).The first pixel circuit may convert the electrical signal into photon counting information, and the second pixel circuit may convert the electrical signal into information for obtaining time of flight (ToF) information.

[0018] The control circuit may switch between the first pixel circuit and the second pixel circuit based on time.

[0019] The control circuit may periodically switch between the first pixel circuit and the second pixel circuit.

[0020] The control circuit may perform control for spatially periodically arranging pixels using the first pixel circuit and pixels using the second pixel circuit in the pixel array.

[0021] The control circuit may perform control for outputting the same type of signals (ie, the first signal or the second signal) from pixels belonging to the same row.

[0022] The control circuit may perform control for outputting different types of signals (ie, first signals or second signals) from pixels belonging to adjacent rows.

[0023] The control circuit may control rows adjacent to a row outputting a first signal so that one row is a row outputting the first signal and the other row is a row outputting a second signal, and may control a row outputting a second signal so that both adjacent rows are rows outputting the first signal.

[0024] The control circuit may divide the pixel array into 2×2 pixel regions and perform control to arrange at least one pixel outputting a first signal and pixels outputting a second signal (except for at least one pixel outputting the first signal) for pixels included in each 2×2 pixel range.

[0025] The control circuit may perform control such that the pixel outputting the second signal is periodically moved in a four-phase cycle.

[0026] The control circuit may divide the pixel array into regions of 3×3 pixels and perform control to arrange at least one pixel outputting a first signal and pixels outputting a second signal (except for at least one pixel outputting the first signal) for pixels included in the range of each 3×3 pixels.

[0027] The control circuit may perform control such that the pixel outputting the second signal is periodically moved in a nine-phase cycle.

[0028] The control circuit may obtain event information based on the second signal in the pixel array, and the control circuit may determine timing of selecting whether each of the plurality of pixels is connected to the first pixel circuit or the second pixel circuit based on the event information.

[0029] The control circuit may obtain event information in a region of the pixel array, and the control unit may select whether pixels belonging to the region are connected to the first pixel circuit or the second pixel circuit based on the event information belonging to the region.

[0030] The control circuit can obtain the optical flow of the moving object existing in the pixel array, the control circuit can set the area in the pixel array based on the optical flow, and the control circuit can select whether the pixels belonging to the area are connected to the first pixel circuit or the second pixel circuit.

[0031] The control circuit may obtain brightness information based on the first signal in the pixel array, and the control circuit may determine timing of selecting whether each of the plurality of pixels is connected to the first pixel circuit or the second pixel circuit based on the brightness information.

[0032] The control circuit may set a region in the pixel array based on the intensity of the brightness information, and the control circuit may select whether pixels belonging to the region are connected to the first pixel circuit or the second pixel circuit.

[0033] The control circuit may set a region in the pixel array based on a result of object recognition using brightness information, and the control circuit may select whether pixels belonging to the region are connected to the first pixel circuit or the second pixel circuit. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a block diagram showing an outline of a solid-state imaging device according to an embodiment.

[0035] Figure 2 is a block diagram showing an outline of a solid-state imaging element according to an embodiment.

[0036] Figure 3 is a diagram showing an overview of a pixel according to an embodiment.

[0037] Figure 4 : is a diagram showing an example of an output arrangement of a first signal and a second signal according to an embodiment.

[0038] Figure 5 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0039] Figure 6 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0040] Figure 7 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0041] Figure 8 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0042] Fig. 9 is a diagram showing an example of a duty ratio of a control signal according to an embodiment.

[0043] Fig.10 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0044] Fig.11 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0045] Fig.12 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0046] Fig.13 is a diagram showing another example of a duty ratio of a control signal according to an embodiment.

[0047] Fig.14 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0048] Fig.15 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0049] Fig.16 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0050] Fig.17 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0051] Fig.18 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0052] Fig.19 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0053] Fig. 20 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0054] Fig.21 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0055] Fig. 22 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0056] Fig.23 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0057] Fig.24 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0058] Fig.25 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0059] Fig.26 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0060] Fig. 27 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0061] Fig.28 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0062] Fig.29 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0063] Fig.30 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0064] Fig.31 is a diagram showing another example of the output arrangement of the first signal and the second signal according to the embodiment.

[0065] Fig.32 is a diagram showing another example of the output arrangement of the first signal and the second signal according to the embodiment.

[0066] Fig.33 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0067] Fig.34 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0068] Fig.35 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0069] Fig.36 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0070] Fig.37 is a diagram showing another example of an output arrangement of a first signal and a second signal according to an embodiment.

[0071] Fig.38 is a block diagram showing an example of a schematic configuration of a vehicle control system.

[0072] Fig.39 1 is an explanatory diagram showing an example of the installation positions of the vehicle exterior information detection unit and the imaging portion. DETAILED DESCRIPTION

[0073] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The accompanying drawings are for explanation, and the shape and size of each component in an actual device, the ratio of the size to other components, etc. are not necessarily as shown in the drawings. In addition, the drawings are shown in a simplified manner, and components necessary for implementation are appropriately provided in addition to those components shown in the drawings.

[0074] Figure 11 is a block diagram showing an overview of a solid-state imaging device according to an embodiment. The solid-state imaging device 1 includes a solid-state imaging element 10, a control unit 16, a storage unit 17, an interface 18, and an optical system 19. The solid-state imaging device 1 includes pixels capable of obtaining multiple pieces of information, outputs multiple pieces of information from multiple pixels at the same time based on various conditions, appropriately performs signal processing, etc., and outputs multiple pieces of information.

[0075] The solid-state imaging element 10 is an element that outputs a signal from a pixel based on light obtained from the outside via the optical system 19. The signal output from the solid-state imaging element 10 is output to the storage unit 17 or the interface 18.

[0076] The control unit 16 is a circuit that controls the solid-state imaging device 1. The control unit 16 controls the solid-state imaging element 10 based on, for example, a command obtained from the outside via the interface 18 or based on a signal obtained from the solid-state imaging element 10. This control may be, for example, control for selecting a signal type to be output from a pixel.

[0077] The storage unit 17 includes a memory, a storage, etc. that temporarily or non-temporarily stores a signal obtained in the solid-state imaging device 1. The information stored in the storage unit 17 can be appropriately output to the outside via the interface 18. In addition, commands, information, etc. from the outside can be stored temporarily or non-temporarily. In the case of specifically realizing information processing by software using hardware resources as at least a part of the processing in the solid-state imaging element 10 or the control unit 16, a program, an execution file, etc. for information processing can be stored in the storage unit 17.

[0078] The interface 18 is an interface for connecting external devices inside and outside the solid-state imaging device 1. The interface 18 may also include a display for displaying images and presenting data to the user, buttons for receiving commands from the user, a touch panel, or a user interface for exchanging information with other users.

[0079] The optical system 19 is an optical system for the solid-state imaging element 10 to appropriately obtain information about light such as light reflected from or transmitted through an imaging target. The optical system 19 may include, for example, one or more lenses, an aperture, a shutter, and the like.

[0080] Figure 2 1 is a block diagram showing an outline of a solid-state imaging element 10 according to an embodiment. The solid-state imaging element 10 includes a pixel array 100, a control circuit 102, a row drive circuit 104, a column drive circuit 106, and a signal processing circuit 108. The solid-state imaging element 10 is, for example, an image sensor that appropriately processes and outputs a signal output from a pixel.

[0081] The pixel array 100 is a region where pixels are arranged in an array. In the pixel array 100, for example, a plurality of pixels are arranged in a two-dimensional array along a first direction and a second direction intersecting the first direction. Here, the first direction may be a row direction, and the second direction may be a column direction.

[0082] The control circuit 102 is a circuit that receives a signal from the control unit 16 and controls each component of the solid-state imaging element 10 so that the pixel array 100 outputs appropriate information. In the following description, each component may perform different processes, which may be controlled by the control circuit 102.

[0083] The row driving circuit 104 is a circuit that selects and outputs a row in the pixel array 100. Furthermore, in the case where a pixel operates as an event detection pixel that performs event detection, the row driving circuit 104 can also operate as an arbitrator.

[0084] The column driver circuit 106 is a circuit that selects a column among pixels belonging to a row selected by the row driver circuit 104 and drives the pixels belonging to the selected column to output from the pixels. Similar to the row driver circuit 104, the column driver circuit 106 may be a circuit that also operates as an arbitrator.

[0085] In addition, the column driver circuit 106 may transmit a signal for switching the output of the pixel to each pixel. For example, in the case where the pixels belonging to a specific area of ​​the pixel array 100 output the second signal and the pixels belonging to another area output the first signal, the pixels belonging to the specific area may be driven by the signal from the column driver circuit 106 to output the second signal. However, the switching of the drive may not be performed by the column driver circuit 106 but by the row driver circuit 104, or as shown by the dotted line, the control circuit 102 may perform the switching of the drive by directly specifying the area.

[0086] The signal processing circuit 108 is a circuit that performs signal processing on the signal output from each pixel of the pixel array 100 and outputs the signal. The signal processing circuit 108 may include, for example, an ADC that converts an analog signal output from the pixel array 100 into a digital signal. In addition, the signal processing circuit 108 may perform signal processing on the digital signal to generate an image signal, or may use an algorithm of target detection, target recognition, etc. on the obtained signal using a model trained by machine learning. The signal processing circuit 108 outputs the processed signal to the outside of the solid-state imaging element 10.

[0087] The signal processing circuit 108 may perform various types of signal processing under the control of the control circuit 102. The control circuit 102 may also send various control signals to various components based on the signal output from the signal processing circuit 108.

[0088] It should be noted that, although not shown, a transient or non-transitory storage area may be provided inside the solid-state imaging element 10. The storage area may be used similarly to the above-described storage unit 17, or may be used as one of various buffers or the like for temporarily storing a signal subjected to signal processing by the signal processing circuit 108.

[0089] Next, the pixel circuit will be described. As described above, the pixel array 100 includes pixels arranged in a two-dimensional array in, for example, the row direction and the column direction.

[0090] Figure 3 1 is a diagram showing an example of a pixel according to an embodiment. The pixel 110 includes a photoelectric conversion element 112, a drive circuit 114, a switch 116, a first pixel circuit 118, and a second pixel circuit 120. In the pixel array 100, Figure 3 The plurality of pixels 110 shown in FIG. 1 are arranged in an array.

[0091] The photoelectric conversion element 112 performs photoelectric conversion on incident light to generate an electric signal. The photoelectric conversion element 112 is a light receiving element determined based on the obtained information.

[0092] In the case where the solid-state imaging device 1 obtains information such as brightness, the photoelectric conversion element 112 may be an element such as a laser diode. In the case where the solid-state imaging device 1 obtains information based on photons, the photoelectric conversion element 112 may be an element such as a single photon avalanche diode (SPAD). As described above, the photoelectric conversion element 112 is appropriately determined according to the purpose of the solid-state imaging device 1.

[0093] The driving circuit 114 is a circuit that drives the photoelectric conversion element 112. When an appropriate signal is input to the driving circuit 114, the photoelectric conversion element 112 converts a signal based on received light into an analog signal and outputs the analog signal.

[0094] The switch 116 is a switch that switches a pixel circuit in which a signal output from the photoelectric conversion element 112 is to be processed before being output from the pixel.

[0095] The first pixel circuit 118 is a circuit that converts the analog signal output from the photoelectric conversion element 112 into a first signal and outputs the first signal. The first pixel circuit 118 can output the first signal obtained as a result of the conversion to the signal processing circuit 108.

[0096] The second pixel circuit 120 is a circuit that converts the analog signal output from the photoelectric conversion element 112 into a second signal and outputs the second signal. The second pixel circuit 120 can output the second signal obtained as a result of the conversion to the signal processing circuit 108.

[0097] Note that each pixel 110 does not necessarily include all components, and some components may be shared with other pixels 110. In the case where there is a floating region in the driving circuit 114, for example, the floating region, the switch 116, the first pixel circuit 118, and the second pixel circuit 120 may be shared. The same applies to other elements, and similar to the elements shared in a general imaging element, as long as the signal processing in the present disclosure is possible, at least a subset of the components may be shared between multiple pixels.

[0098] The driving of the photoelectric conversion element 112 by the driving circuit 114 of the pixel 110 and the switching by the switch 116 can be controlled by the control circuit 102. In the pixel array 100, the same control can be performed on any number of pixels 110 (that is, for example, on the pixels 110 in any area). In addition, the control circuit 102 can control in each pixel whether to output the first signal via the first pixel circuit 118 or the second signal via the second pixel circuit 120. For example, the control signal can be transmitted to the pixel 110 via the column driving circuit 106, or can be transmitted to the pixel 110 via another path.

[0099] In other words, the control circuit 102 may control whether the first signal or the second signal is output for each pixel 110 belonging to the pixel array 100 .

[0100] The control circuit 102 may also switch between outputting a first signal via a first pixel circuit and outputting a second signal via a second pixel circuit based on the arrangement of each pixel 110 in the pixel array 100. That is, the control circuit 102 may spatially specify and switch which signal is to be output from which pixel 110 in the pixel array 100.

[0101] In addition, the control circuit 102 can also control the timing of outputting the first signal using the first pixel circuit and the timing of outputting the second signal using the second pixel circuit for any pixel 110 belonging to the pixel array 100. That is, the control circuit 102 can temporarily switch which signal is to be output for the pixel 110 in the pixel array 100.

[0102] To perform such control, the control circuit 102 may output control signals to components that need to be controlled in the pixel 110. As an example, the control circuit 102 may output a drive control signal for the drive circuit 114, a switching control signal for the switch 116, a drive control signal for the first pixel circuit 118, and a drive control signal for the second pixel circuit 120 to the components in each pixel 110 via appropriate paths.

[0103] As a non-limiting example, the photoelectric conversion element 112 may be an element that obtains a brightness signal. In this case, the first pixel circuit 118 and the second pixel circuit 120 may convert a signal related to brightness and output the signal. For example, the first pixel circuit 118 may convert brightness information, which is an analog electrical signal obtained by the photoelectric conversion element 112, into a grayscale signal and output the grayscale signal as a first signal, and the second pixel circuit 120 may read a change in brightness information obtained by the photoelectric conversion element 112, convert the change into event detection information, and output the event detection information as a second signal.

[0104] As another non-limiting example, the photoelectric conversion element 112 may be an element capable of receiving light in photon units. In this case, the first pixel circuit 118 and the second pixel circuit 120 may convert a signal related to information obtained by sensing photons and output the converted signal. For example, the first pixel circuit 118 may convert the electrical signal obtained by the photoelectric conversion element 112 into photon counting information, and the second pixel circuit 120 may convert the electrical signal obtained by the photoelectric conversion element 112 into information for obtaining time of flight (ToF) information and output the information. In this case, the negative voltage applied to the photoelectric conversion element may be controlled together with the switching between the first pixel circuit 118 and the second pixel circuit 120.

[0105] In addition, when different types of signals can be obtained from signals that can be obtained according to the purpose, the photoelectric conversion element 112 can use any element that obtains such a signal. In addition, according to the selection of elements, a plurality of pixel circuits can be set to any pixel circuit that can obtain different types of signals.

[0106] As some embodiments, a combination of the photoelectric conversion element 112, the first pixel circuit 118, and the second pixel circuit 120 and a spatial and temporal control method will be described with reference to examples. In addition, the pixel 110 in each embodiment is described using a pixel that obtains brightness and a pixel that detects an event, but is not limited to this, and any pixel may be switched as long as the first signal and the second signal are appropriately obtained.

[0107] (First Embodiment)

[0108] Figure 4 and Figure 5 1 and 2 are diagrams showing examples of conversion of output arrangements of first and second signals in a pixel array 100 according to an embodiment. These drawings show some pixels 110 of the pixel array 100. It should be noted that in the following drawings, the first direction and the second direction are shown as non-limiting examples, and the present disclosure according to the embodiment does not exclude directions other than these directions.

[0109] In the pixels 110, R indicates a pixel for obtaining red brightness information, G indicates a pixel for obtaining green brightness information, and B indicates a pixel for obtaining blue brightness information. These pixels 110 are connected to a first pixel circuit 118, and output a brightness signal based on the intensity of incident light as a first signal.

[0110] On the other hand, EV represents a pixel that outputs event detection information. Such a pixel 110 is connected to the second pixel circuit 120, and outputs an event detection signal for obtaining a change in the intensity of incident light as a second signal.

[0111] Note that the configuration of pixels for obtaining brightness information may be a Bayer array as shown, or may include pixels that receive light of other colors or wavelengths such as white (W) or cyan (Cy). As another example, pixels that receive infrared (IR) light, pixels that include plasma filters, etc. may be provided. In addition, pixels for another purpose, such as pixels in different polarization states or pixels for obtaining image plane phase differences, may be provided.

[0112] like Figure 4 As shown, as the first phase, for example, the control circuit 102 performs control for outputting the first signal from the odd-numbered rows and performs control for outputting the second signal from the even-numbered rows.

[0113] like Figure 5 As shown, as the second phase, for example, the control circuit 102 performs control for outputting the second signal from the odd-numbered rows and performs control for outputting the first signal from the even-numbered rows.

[0114] As described above, the control circuit 102 may perform control such that the pixels 110 outputting the first signal and the second signal (ie, pixels outputting signals via the first pixel circuit 118 and pixels outputting signals via the second pixel circuit 120 ) are spatially periodically arranged.

[0115] In addition, the control circuit 102 may perform control for switching periodically in time, that is, control for switching at a periodic timing, Figure 4 The arrangement in the first phase shown in Figure 5 As described above, the control circuit 102 can obtain information based on the first signal and information based on the second signal by spatially and temporally switching information related to light received by the photoelectric conversion element without reducing resolution.

[0116] Note that, as another example, the control circuit 102 may provide a timing at which all pixels included in the pixel array 100 output the first signal between the first phase and the second phase, for example. This is the same as the following modes.

[0117] Despite Figure 4 and Figure 5 The two-phase transition is performed in , but the transition can be performed in more phases than two. Figures 6 to 13 The three-phase transformation is described in Figures 13 to 17 The four-phase transition is described in Fig.18 Nine phase transitions are described in , but the type of transition is not limited thereto, and transitions of any number of phases may be achieved if high resolution information can be appropriately obtained.

[0118] Figure 6 , Figure 7 and Figure 8 is a diagram showing an example of three-phase transition.

[0119] like Figure 6 As shown, as the first phase, for example, the control circuit 102 performs control for outputting the first signal from the (3n+1)th row and the (3n+2)th row, and performs control for outputting the second signal from the (3n)th row.

[0120] like Figure 7 As shown, as the second phase, for example, the control circuit 102 performs control for outputting the first signal from the (3n+2)th row and the (3n)th row, and performs control for outputting the second signal from the (3n+1)th row.

[0121] like Figure 8 As shown, as the third phase, for example, the control circuit 102 performs control for outputting the first signal from the (3n+1)th row and the (3n)th row, and performs control for outputting the second signal from the (3n+2)th row.

[0122] The control circuit 102 may thus perform control for periodically arranging the pixels 100 outputting the first signal and the second signal in a set of three rows. Alternatively, the control circuit 102 may perform control for periodically switching the first phase to the third phase in time.

[0123] Fig. 9 is a diagram showing an example of control signals output from the control circuit 102 to the respective rows. By receiving such control signals, in the first phase, the (3n)th row outputs the second signal, and the other rows output the first signal. In the second phase, the (3n+1)th row outputs the second signal, and the other rows output the first signal. In the third phase, the (3n+2)th row outputs the second signal, and the other rows output the first signal.

[0124] Therefore, the time density of the pixel 110 outputting the second signal can be set based on the duty cycle. Specifically, when the number of phases increases to three or more, the control circuit 102 can set the time density of the pixel 110 outputting the second signal by controlling the duty cycle. For example, as described below, the time density of outputting the second signal can be increased.

[0125] Fig.10 , Fig.11 and Fig.12 is a diagram showing another example of three-phase transition.

[0126] like Fig.10 As shown, as the first phase, for example, the control circuit 102 performs control for outputting the first signal from the (3n+1)th row, and performs control for outputting the second signal from the (3n+2)th row and the (3n)th row.

[0127] like Fig.11 As shown, as the second phase, for example, the control circuit 102 performs control for outputting the first signal from the (3n+2)th row, and performs control for outputting the second signal from the (3n+1)th row and the (3n)th row.

[0128] like Fig.12 As shown, as the third phase, for example, the control circuit 102 performs control for outputting the first signal from the (3n)th row, and performs control for outputting the second signal from the (3n+1)th row and the (3n+2)th row.

[0129] As shown in these figures, the control circuit 102 can perform control for arranging more pixels 110 that output the second signal than pixels that output the first signal.

[0130] Fig.13 It is shown in Figures 10 to 12 1 is a diagram showing an example of a control signal output from the control device 102 to each row in the case of . By receiving such a control signal, it is possible to execute Figures 10 to 12 control.

[0131] and Fig. 9 In contrast, the control circuit 102 may indicate the number of pixels outputting the first signal or the second signal at different duty ratios to determine which of the signals of the more pixels 110 is to be output.

[0132] Fig.14 , Fig.15 , Fig.16 and Fig.17 is a diagram showing an example of a four-phase transition. As an example of performing a four-phase transition, the control circuit 102 may control the signals output to the pixels 110 in the pixel array 100 in 2×2 blocks in space and time.

[0133] like Fig.14 As shown in , as a first phase, the control circuit 102 performs control to output the second signal from the pixel 110 at the lower right position of every 2×2 pixels in the pixel array 100 and to output the first signal from the other pixels.

[0134] like Fig.15 As shown, as the second phase, the control circuit 102 performs control to output the second signal from the pixel 110 at the upper left position of every 2×2 pixels in the pixel array 100 and to output the first signal from the other pixels.

[0135] like Fig.16 As shown, as the third phase, the control circuit 102 performs control to output the second signal from the pixel 110 at the upper right position of every 2×2 pixels in the pixel array 100 and to output the first signal from the other pixels.

[0136] like Fig.17 As shown, as the fourth phase, the control circuit 102 performs control to output the second signal from the pixel 110 at the lower left position of every 2×2 pixels in the pixel array 100 and to output the first signal from the other pixels.

[0137] For example, the first phase to the fourth phase may be switched periodically in time. Four-phase control may also be performed in this manner. In the case of performing this control, for example, the pixel 110 may be controlled while sharing the reference voltage applied to obtain the second signal between each 2×2 pixels, or each pixel may be controlled uniquely.

[0138] For example, a capacitor that maintains a reset voltage can be shared between every 2×2 pixels. By sharing circuits in this way, some processing (i.e., event detection processing) can also be implemented, for example, based on a comparison result between a voltage reset in another layer and an output voltage of a pixel in the layer that generates a second output. In the case where grayscale information is output as a first signal and event detection information is output as a second signal, by sharing a capacitor, the second pixel circuit 120 that outputs event detection information, whose circuit area tends to be larger than the circuit area of ​​the first pixel circuit 118 that outputs grayscale information, can be reduced, and the sensor as a whole can also be miniaturized.

[0139] Fig.18 This is an extension from division in units of 2×2 pixels to division in units of 3×3 pixels. In this case, the control circuit 102 can output the second signal appropriately by, for example, periodically controlling nine phases.

[0140] For example, the control circuit 102 can control the first phase for outputting the second signal from the upper left, the second phase for outputting the second signal from the center of the top row, ..., and the ninth phase for outputting the second signal from the lower right in a periodic timing manner in the case of pixels divided into 3×3 units.

[0141] The timing of outputting the second signal may be measured, and the sensor may be reduced similarly to the above.

[0142] As described above, with the solid-state imaging device according to the present embodiment, the pixels that output the second signal can be periodically controlled in space and time. By periodically controlling the pixels belonging to the pixel array in space and time, the solid-state imaging device can obtain the output of two types of signals for the entire image to be captured at a high resolution and a high frame rate.

[0143] (Second Embodiment)

[0144] In the first embodiment, for the pixels 110 belonging to the pixel array 100, the pixels 110 outputting the first signal and the pixels 110 outputting the second signal are arranged in the same spatial period. The arrangement of the pixels is not limited thereto, but an arrangement of different signal outputs may be set in a partial area of ​​the pixel array 100.

[0145] Fig.19 , Fig. 20 and Fig.21 is a diagram showing an example of a configuration of a signal output of the pixel 110 according to the present embodiment. Note that in the following drawings, as an example, the pixel 110 outputting the second signal is represented by "2", and the pixel outputting the first signal is represented by a blank.

[0146] like Fig.19 As shown, as a first phase, the control circuit 102 controls the pixels 110 belonging to the area other than the area ROI of the pixel array 100 so that Figure 6 As shown in , the second signal is output from one of every three rows, and the first signal is output from the other two rows. Fig.10 As shown, the control circuit 102 controls the pixels 110 belonging to the area ROI of the pixel array 100 so that the second signal is output from two of every three rows and the first signal is output from the other row.

[0147] like Fig. 20 As shown in FIG. 1 , as the second phase, the control circuit 102 controls the pixels 110 belonging to the area other than the area ROI of the pixel array 100 so that Figure 7 As shown, the second signal is output from one of every three rows, and the first signal is output from the other two rows. On the other hand, the control circuit 102 controls the pixels 110 belonging to the area ROI of the pixel array 100 so that Fig.11 As shown, the second signal is output from two rows out of every three rows and the first signal is output from the other row.

[0148] like Fig.21 As shown in FIG. 1 , as the third phase, the control circuit 102 controls the pixels 110 belonging to the area other than the area ROI of the pixel array 100 so that Figure 8 As shown in , the second signal is output from one of every three rows, and the first signal is output from the other two rows. On the other hand, the control circuit 102 controls the pixels 110 belonging to the region ROI of the pixel array 100 so that as shown in Fig.12 As in the example, the second signal is output from two of every three rows and the first signal is output from the other row.

[0149] Therefore, the pixel array 100 can be divided into a plurality of areas in units of rows, and signals with different duty ratios can be obtained. In the case where it is desired to detect an event in a nearby object or accurately measure the distance to an object in an area close to the vehicle body as in the case of a vehicle-mounted camera, for example, the solid-state imaging device 1 can obtain an appropriate signal with a high density on the side of the pixel array 100 closer to the vehicle body.

[0150] Fig. 22 , Fig.23 , Fig.24 as well as Fig.25 100. As shown in these drawings, the region ROI may be set at any position in the pixel array 100.

[0151] like Fig. 22 In the first phase shown in FIG. 1 , the control circuit 102 may control pixels 110 belonging to an area of ​​the pixel array 100 other than the area ROI in such a way that Fig.14 On the other hand, for the pixels 110 belonging to the area ROI, the control circuit 102 may output the second signal from the upper left and lower right pixels 110 and the first signal from the upper right and lower left pixels 110 in each block including 2×2 pixels.

[0152] like Fig.23 As shown in FIG. 1 , as a second phase, the control circuit 102 may control the pixels 110 belonging to the area of ​​the pixel array 100 other than the area ROI in such a manner that, as shown in FIG. Fig.17As shown in , the second signal is output from one of the 2×2 pixels included in each block, and the first signal is output from the other three pixels. On the other hand, for the pixels 110 belonging to the area ROI, the control circuit 102 may output the second signal from the upper right and lower left pixels 110 and the second signal from the upper left and lower right pixels 110 in each block including the 2×2 pixels.

[0153] like Fig.24 As shown, as the third phase, the control circuit 102 can control the pixels 110 belonging to the area other than the area ROI of the pixel array 100, so that Fig.15 As shown, the second signal is output from one of the 2×2 pixels included in each block, and the first signal is output from the other three pixels. On the other hand, for the pixels 110 belonging to the area ROI, the control circuit 102 may output the second signal from the upper left and lower right pixels 110 and the first signal from the upper right and lower left pixels 110 in each block including 2×2 pixels.

[0154] like Fig.25 As shown in the fourth phase, the control circuit 102 may control the pixels 110 belonging to the area other than the area ROI of the pixel array 100 so that Fig.16 As shown, the second signal is output from one of the 2×2 pixels included in each block, and the first signal is output from the other three pixels. On the other hand, for the pixels 110 belonging to the area ROI, the control circuit 102 may output the second signal from the upper right and lower left pixels 110 and the second signal from the upper left and lower right pixels 110 in each block including the 2×2 pixels.

[0155] As described above, with the solid-state imaging device according to the present embodiment, the region ROI can be set not for each row but for any set of pixels 110. Figure 22 to Figure 25 As can be seen from the example of , the number of phases may be different between the inside and outside of the region ROI. Although four phases and two phases are mixed in the above example, other numbers of phases, such as three phases and two phases, may be mixed instead. For example, in Fig.19 In each row of ROI, Fig. 22 In an ROI of, for example, the control circuit 102 may also perform control for outputting the first signal at a density of 1 / 3 and outputting the second signal at a density of 2 / 3.

[0156] The above is an example of setting the region ROI, and the position, shape, number, etc. of the region ROI can be set as needed. In addition, the region ROI does not need to be fixed, and its position, shape, number, etc. can be changed at appropriate timing according to the situation.

[0157] (Third Embodiment)

[0158] Although the setting of the fixed region ROI has been described in the second embodiment, in this embodiment, it will be described that the control circuit 102 changes the position, shape, etc. of the region ROI as necessary.

[0159] Fig.26 The solid-state imaging device 1 can detect movement of the object OBJ based on the result of processing the obtained grayscale information as the first signal or the obtained event detection information as the second signal by the signal processing circuit 108.

[0160] When using grayscale information, the signal processing circuit 108 can detect the object OBJ by any object recognition method, for example. In addition, the signal processing circuit 108 can simply determine that the object is moving in an area where the grayscale change is large, and notify the control circuit 102 of the determination result or an area that can be obtained from the determination result.

[0161] For example, in the case of using event detection information, the signal processing circuit 108 may determine that an object is operating in a region including the pixel 110 in which an event has been detected.

[0162] In other words, the control circuit 102 can predict an event that will occur based on the output result of the first signal or the second signal in the pixel array 100, determine the timing of selecting whether each of the multiple pixels 110 belonging to the pixel array 100 is connected to the first pixel circuit 118 or the second pixel circuit 102 based on the prediction result, and control the arrangement of the pixels 110 that output the second signal.

[0163] Specifically, the control circuit 102 may obtain an event detection signal as the second signal, set a region based on the event information, and perform control for switching whether the pixels 110 in the region output the first signal or the second signal.

[0164] The control circuit 102 may calculate an optical flow from the movement of the object OBJ, and set the arrangement of the pixels 110 from which the second signal is obtained based on the optical flow.

[0165] If it is estimated as a result of obtaining the optical flow that the object OBJ will move in the direction indicated by the arrow in the figure, the control circuit 102 performs control for arranging the pixels 110 that output the second signal based on the estimated movement direction.

[0166] Fig. 27 Shown in Fig.261 is an example of arrangement of pixels 110 that output the second signal based on the moving direction and speed of the object estimated from the optical flow in the state of FIG. As shown in the figure, the control circuit 102 can perform control for arranging the pixels 110 as the pixels 110 that output the second signal in the moving direction estimated from the optical flow. As the area to be switched, for example, the control circuit 102 can set a predetermined area in front of the moving direction of the object OBJ, and can perform control for switching between outputting the first signal and outputting the second signal in the predetermined area.

[0167] Fig.28 It is shown that in obtaining Fig. 27 FIG. 1 is a diagram showing an example of a configuration of a pixel 110 that outputs a second signal after an optical flow in FIG. Fig. 27 and Fig.28 As shown in , the control circuit 102 may perform control for arranging the pixels 110 that output the second signal so that the movement of the object OBJ can be captured in a downward direction in which the movement of the object OBJ is estimated.

[0168] The control circuit 102 can thus obtain the optical flow of the object present in the pixel array 100. The control circuit 102 can set the area in the pixel array 100 based on the optical flow, and perform control for selecting whether to output the first signal or the second signal from the pixels 110 belonging to the area.

[0169] The region may be set to a region including the object OBJ and a margin provided in the moving direction of the object OBJ estimated from the optical flow. The control circuit 102 may perform control for switching all pixels 110 in the region from pixels outputting the first signal to pixels outputting the second signal. As another example, the control circuit 102 may switch at least a subset of the pixels 110 from pixels outputting the first signal to pixels outputting the second signal, rather than all pixels 110 belonging to the region.

[0170] As described above, using the solid-state imaging device according to the present embodiment, a moving object can be detected and the pixels outputting the second signal can be arranged based on the detection result. By arranging the pixels outputting the second signal in this way, the object can be accurately tracked and the first signal (e.g., grayscale information) with a higher resolution and a higher frame rate can be obtained at a position where there is no movement.

[0171] (Fourth Embodiment)

[0172] In this embodiment, another example of extraction of the region where the second signal is output and arrangement of pixels will be described.

[0173] Fig.291 is a diagram showing an example of the arrangement of pixels that output the first signal and the second signal when there are pixels 110 with saturated luminance values ​​in the pixel array 100. For example, the saturation area SAT is an area where the luminance value is saturated. Although the saturation area SAT is represented as a rectangular area here, the shape and size are not limited thereto. For example, in an image captured by a vehicle-mounted camera, an area of ​​saturated pixels 110 is sometimes obtained at an exit of a tunnel or the like.

[0174] In pixels with saturated brightness values, it is difficult to properly obtain grayscale information. In an area representing such grayscale values, pixels that output the second signal instead of the first signal can be properly arranged. With such a configuration, event information can be properly obtained in an area where grayscale information cannot be properly obtained.

[0175] Fig.30 is a diagram showing another example according to the present embodiment. As shown in the figure, the control circuit 102 may arrange (i.e., periodically arrange) for example, pixels that output the first signal and pixels that output the second signal for the pixels 110 belonging to the saturation region SAT. Similar to the above embodiment, in Fig.30 In the setting, control for periodically switching between the two phases can be performed.

[0176] Furthermore, the control circuit 102 may also perform control with appropriate switching timing and arrangement of four phases, nine phases, etc. instead of two phases.

[0177] As described above, with the solid-state imaging device according to the present embodiment, the second signal can be appropriately obtained by appropriately arranging pixels that output the second signal in the region where the value of the first signal is saturated. For example, if an event occurs in a saturated region when the first signal is a grayscale signal and the second signal is an event detection signal, the solid-state imaging device 1 can obtain grayscale values ​​with high accuracy and high resolution in other regions and quickly respond to the detection of the event.

[0178] (Fifth Embodiment)

[0179] In this embodiment, another example of extraction and setting of regions will be described.

[0180] Fig.31 1 is a diagram showing a mode in which the pixel 110 belonging to the pixel array 100 obtains a grayscale value as a first signal. The signal processing circuit 108 detects a target such as a person based on grayscale information. The target is not limited to a person and may be any object.

[0181] The control circuit 102 extracts the region ROI where the target is detected, and arranges the pixels 110 that output the second signal in the region ROI. Note that the region ROI may have a rectangular shape as shown in the figure, or may have other shapes.

[0182] Fig.32 1 is a diagram showing an example in which the pixels 110 outputting the second signal are arranged in the region ROI. For example, the control circuit 102 may perform control for arranging the pixels 110 outputting the second signal undergoing four-phase conversion. The signal processing circuit 108 or an external or internal processing circuit may implement tracking of a target, etc. based on the event detection information output from the pixel array 100.

[0183] Fig.33 1 is a diagram showing another example in which the pixels 110 outputting the second signal are arranged in the region ROI. For example, the control circuit 102 may perform control for arranging the pixels 110 outputting the second signal undergoing two-phase conversion. Fig.32 , the solid-state imaging device 1 can realize tracking of a target etc. based on event detection information.

[0184] Notice, Fig.32 and Fig.33 The modes shown in are not limited to being applied alone. Fig.31 As shown, the solid-state imaging device 1 can detect the target and set the area ROI, and then Fig.32 As shown, pixels 110 that perform four-phase transitions are arranged.

[0185] In this state, when the movement becomes smaller, it can be established Fig.31 On the other hand, in Fig.32 When the number of pixels 110 detecting events increases in the region ROI, it can be achieved Fig.33 The control is performed in the manner shown in the four-phase conversion arrangement.

[0186] In other words, as an example, the solid-state imaging device 1 can obtain grayscale information as a first signal in the pixel array 100, and determine the timing of selecting whether each of the multiple pixels 110 belonging to the pixel array 100 is connected to the first pixel circuit 118 or the second pixel circuit 120 based on the grayscale information.

[0187] The control circuit 102 may set the area ROI based on the change in the intensity of the grayscale information. In addition, the control circuit 102 may select whether the pixel 110 in the area ROI is connected to the first pixel circuit 118 or the second pixel circuit 120.

[0188] The solid-state imaging device 1 can also set the area ROI based on the result of target recognition or object recognition obtainable from the grayscale information.

[0189] As described above, with the solid-state imaging device 1 according to the present embodiment, the region ROI can be set based on the event detection information obtained as the second signal, and the density in the region ROI for outputting the second signal can be changed based on the event detection information in the set region ROI. With this control, the solid-state imaging device 1 can obtain event detection information with high accuracy and high resolution, and at the same time obtain grayscale information with high resolution.

[0190] (Sixth Embodiment)

[0191] The solid-state imaging device 1 in the present disclosure can also be applied to eye tracking technology of smart glasses, etc. In smart glasses, etc., the solid-state imaging device 1 can capture images from the front side of the eye, or can capture images of reflected light by reflecting infrared light using a grating formed in a hologram, etc. provided in the glasses.

[0192] Fig.34 1 is a diagram showing an example of obtaining an image of an eye in the pixel array 100. The solid-state imaging device 1 can also receive reflected light by irradiating the eye using an infrared light source provided inside or outside the solid-state imaging device 1. When infrared light is used in this manner, the reflected light from the retina forms a Purkinje image arranged at a predetermined position relative to the eye contour. For example, the Purkinje image is formed at the position of the pixel 110 indicated by the hatched line in the figure.

[0193] Note that, considering easy viewing of the drawings, four pixels 110 are arranged at the position where the Purkinje image is generated, and the arrangement of the pixels 110 is not limited to this. For example, the pixels 110 may be arranged more densely, in which case more pixels 110 are used as pixels 110 indicating the position of the Purkinje image. In addition, the solid-state imaging device 1 can obtain information about incident light related to the position of the Purkinje image in four or fewer pixels 110.

[0194] The control circuit 102 can control the pixel 110 arranged at the position of the Purkinje image as a pixel that obtains grayscale information as the first signal. With this arrangement, for example, in the case where smart glasses or the like are displaced relative to the eyes, the signal processing circuit 108 or the like can detect that the Purkinje image is no longer available, and can arrange the pixel 110 that obtains grayscale information at the position of a new Purkinje image. By determining the position of the pixel 110 that obtains the Purkinje image, the solid-state imaging device 1 can also obtain information about the eye area, the area where the pupil moves, and the like.

[0195] The control circuit 102 may set a first signal for obtaining grayscale information at the pixel 110 arranged at the position of the Purkinje image, and may also control a second signal for obtaining event detection information in an appropriate arrangement to improve the accuracy of eye tracking.

[0196] Fig.35 is a diagram showing an example of arrangement of pixels that obtain a first signal and pixels that are a second signal in the pixel array 100. Similar to the above, "2" in a pixel indicates a pixel that obtains a second signal.

[0197] As shown in the figure, the control circuit 102 can determine the position of the pupil and arrange the pixel 110 that outputs the second signal at the position of the pupil. The pupil is a part of the eye that moves a lot, and by arranging the pixels that output the second signal in this way, the accuracy of eye tracking for obtaining the position of the pupil (i.e., the direction of the line of sight) can be improved.

[0198] In addition, the control circuit 102 may also perform iris recognition based on grayscale information by arranging pixels 110 that output the first signal for the iris region around the pupil.

[0199] Fig.36 1 is a diagram showing the arrangement of pixels in the case of different positions of the pupil. As shown in the figure, the control circuit 102 can arrange the pixels 110 that obtain the second signal as event detection information based on the position of the pupil. As shown in the figure, regardless of whether the pixel overlaps with the pupil, the pixel 110 that obtains the first signal is arranged as the pixel 110 indicated by the hatched line at the position of the Purkinje image.

[0200] Note that, in the case of pixels 110 capable of obtaining a higher resolution image, the control circuit 102 may also set any number of pixels 110 around the pupil as pixels 110 for obtaining the second signal, in which case the movement of the pupil may be tracked more accurately.

[0201] In addition, if Fig.37 As shown, the control circuit 102 may further arrange the area around the aperture as the pixels for obtaining the second signal 110. By arranging the event detection pixels in this manner, the movement of the pupil may be detected more accurately.

[0202] As described above, using the solid-state imaging device according to the present embodiment, for example, the Purkinje image is detected in eye tracking to appropriately determine the area of ​​the eye in the pixel array, the event detection pixels are arranged at the position of the pupil, so that the movement of the pupil can be accurately tracked, and human authentication, etc. can be accurately achieved from the grayscale information about the iris.

[0203] The solid-state imaging element 10 in the present disclosure can be mounted as a chip as necessary. The chip can be realized by stacking a plurality of semiconductor layers on each other.

[0204] For example, pixels and other circuits may be formed on each of two semiconductor substrates, and these substrates may be stacked on each other by any method (e.g., CoC: chip on chip, CoW: chip on wafer, or WoW: wafer on wafer). At least a light receiving element (photoelectric conversion element) may be provided in the upper layer, and other circuits may be appropriately allocated to the upper and lower layers. These layers may be electrically connected to each other by any suitable method (e.g., through-holes or micro-soldering).

[0205] Note that the number of layers does not have to be two, but pixels and other circuits may be formed on one semiconductor substrate or may be formed on three or more semiconductor substrates.

[0206] The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can also be implemented as a device installed on any kind of mobile body such as a car, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot, a construction machine, an agricultural machine (tractor), etc.

[0207] Fig.38 7000 is a block diagram showing a schematic configuration example of a vehicle control system 7000 as an example of a mobile body control system to which the technology of the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected to each other via a communication network 7010. Fig.38 In the example shown, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. For example, the communication network 7010 connecting the plurality of control units to each other may be an in-vehicle communication network conforming to any standard, such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), FlexRay (registered trademark), and the like.

[0208] Each control unit includes: a microcomputer that performs calculations according to various programs; a storage unit that stores programs executed by the microcomputer, parameters for various operations, etc.; and a drive circuit that drives various control object devices. Each control unit also includes: a network interface (I / F) for communicating with other control units via the communication network 7010; and a communication I / F for communicating with devices, sensors, etc. inside and outside the vehicle through wired communication or radio communication. Fig.38, a microcomputer 7610, a general communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an onboard device I / F 7660, a sound / image output unit 7670, an onboard network I / F 7680, and a storage unit 7690 are shown as functional components of the integrated control unit 7600. Other control units similarly include a microcomputer, a communication I / F, a storage unit, and the like.

[0209] The drive system control unit 7100 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 7100 is used as a control device for a drive force generating device (such as an internal combustion engine, a drive motor, etc.) for generating the drive force of the vehicle, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, etc. The drive system control unit 7100 may have a function as a control device for an anti-lock braking system (ABS), an electronic stability control (ESC), etc.

[0210] The drive system control unit 7100 is connected to the vehicle state detection unit 7110. The vehicle state detection unit 7110 includes, for example, a gyro sensor for detecting the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor for detecting the acceleration of the vehicle, and at least one of sensors for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine speed, or the speed of the wheels. The drive system control unit 7100 performs calculation processing using the signal input from the vehicle state detection unit 7110 to control the internal combustion engine, the drive motor, the electric power steering device, the brake device, and the like.

[0211] The body system control unit 7200 controls the operation of various devices provided in the body according to various programs. For example, the body system control unit 7200 is used as a control device for a keyless entry system, a smart key system, a power window device, or various lights such as a headlight, a backup light, a brake light, a turn signal, a fog light, etc. In this case, a radio wave emitted from a mobile device may be input to the body system control unit 7200 as a substitute for a signal of a key or various switches. The body system control unit 7200 receives these input radio waves or signals, and controls a door lock device, a power window device, a light, etc. of the vehicle.

[0212] The battery control unit 7300 controls the secondary battery 7310 as a power source for driving the motor according to various programs. For example, information about the battery temperature, the battery output voltage, the amount of charge remaining in the battery, etc. is supplied to the battery control unit 7300 from the battery device including the secondary battery 7310. The battery control unit 7300 performs arithmetic operation processing using these signals, and performs control for adjusting the temperature of the secondary battery 7310 or controls a cooling device provided to the battery device, etc.

[0213] The vehicle exterior information detection unit 7400 detects information outside the vehicle including the vehicle control system 7000. For example, the vehicle exterior information detection unit 7400 is connected to at least one of the imaging unit 7410 and the vehicle exterior information detection unit 7420. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. For example, the vehicle exterior information detection unit 7420 includes at least one of an environmental sensor for detecting current atmospheric conditions or weather conditions and a peripheral information detection sensor for detecting other vehicles, obstacles, pedestrians, etc., outside the vehicle including the vehicle control system 7000.

[0214] For example, the environmental sensor may be at least one of a raindrop sensor for detecting rain, a fog sensor for detecting fog, a sunlight sensor for detecting the degree of sunlight, and a snow sensor for detecting snowfall. The peripheral information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR device (a light detection and ranging device, or a laser imaging detection and ranging device). Each of the imaging unit 7410 and the vehicle exterior information detection unit 7420 may be provided as an independent sensor or device, or may be provided as a device in which a plurality of sensors or devices are integrated.

[0215] Here, Fig.39 An example of the installation position of the imaging unit 7410 and the vehicle exterior information detection unit 7420 is shown. The imaging units 7910, 7912, 7914, 7916 and 7918 are, for example, arranged at least one of the positions on the front nose, side mirrors, rear bumper and rear door of the vehicle 7900 and the position on the upper part of the windshield in the vehicle interior. The imaging unit 7910 arranged at the front nose of the vehicle interior and the imaging unit 7918 arranged at the upper part of the windshield mainly obtain images in front of the vehicle 7900. The imaging units 7912 and 7914 arranged at the side mirrors mainly obtain images of the sides of the vehicle 7900. The imaging unit 7916 arranged at the rear bumper or rear door mainly obtains images of the rear of the vehicle 7900. The imaging unit 7918 arranged at the upper part of the windshield in the vehicle interior is mainly used to detect the front vehicle, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0216] It should be noted that Fig.39An example of the imaging range of each imaging unit 7910, 7912, 7914, and 7916 is described. Imaging range a represents the imaging range of the imaging unit 7910 set to the front nose. Imaging ranges b and c respectively represent the imaging ranges of the imaging units 7912 and 7914 provided to the side mirrors. Imaging range d represents the imaging range of the imaging unit 7916 set to the rear bumper or rear door. For example, by superimposing image data imaged by the imaging units 7910, 7912, 7914, and 7916, a bird's-eye view image of the vehicle 7900 viewed from above can be obtained.

[0217] The vehicle exterior information detection units 7920, 7922, 7924, 7926, 7928, 7930 provided at the front, rear, side, corners, and upper portion of the windshield inside the vehicle 7900 may be, for example, ultrasonic sensors or radar devices. The vehicle exterior information detection units 7920, 7926, 7930 provided at the front nose, rear bumper, rear door of the vehicle 7900, and upper portion of the windshield inside the vehicle may be, for example, LIDAR devices. These vehicle exterior information detection units 7920 to 7930 are mainly used to detect preceding vehicles, pedestrians, obstacles, and the like.

[0218] Return to reference Fig.38 , will continue to describe. The vehicle exterior information detection unit 7400 enables the imaging unit 7410 to capture images outside the vehicle and receive captured image data. In addition, the vehicle exterior information detection unit 7400 receives detection information from the vehicle exterior information detection unit 7420 connected to the vehicle exterior information detection unit 7400. In the case where the vehicle exterior information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the vehicle exterior information detection unit 7400 sends ultrasonic waves, electromagnetic waves, etc., and receives information about the reflected waves. Based on the received information, the vehicle exterior information detection unit 7400 can perform processing for detecting objects such as humans, vehicles, obstacles, signs, and text on the road surface, or for detecting the distance between objects. In addition, the vehicle exterior information detection unit 7400 can also perform environmental recognition processing for identifying rainfall, fog, road conditions, etc. based on the received information. The vehicle exterior information detection unit 7400 can calculate the distance to objects outside the vehicle based on the received information.

[0219] In addition, the vehicle exterior information detection unit 7400 can perform image recognition processing to identify humans, vehicles, obstacles, signs, text on the road surface, etc., or detect the distance thereto based on the received image data. The vehicle exterior information detection unit 7400 can perform processing such as distortion correction, alignment, etc. on the received image data, and combine the image data imaged by multiple different imaging units 7410 to generate a bird's-eye view image or a panoramic image. The vehicle exterior information detection unit 7400 can perform viewpoint transformation processing using the image data captured by the imaging unit 7410, which includes different imaging units.

[0220] The vehicle-mounted information detection unit 7500 detects information about the interior of the vehicle. For example, the vehicle-mounted information detection unit 7500 is connected to a driver state detection unit 7510 that detects the state of the driver. The driver state detection unit 7510 may include a camera that images the driver, a biosensor that detects the driver's biometric information, a microphone that collects sounds inside the vehicle, and the like. The biosensor is, for example, arranged in a seat surface, a steering wheel, and the like, and detects biometric information of a passenger sitting on the seat or a driver holding the steering wheel. Based on the detection information input from the driver state detection unit 7510, the vehicle-mounted information detection unit 7500 can calculate the driver's fatigue or the driver's concentration, or can determine whether the driver is dozing off. The vehicle-mounted information detection unit 7500 may subject the audio signal obtained by collecting the sound to processing such as noise elimination processing.

[0221] The integrated control unit 7600 controls the general operation within the vehicle control system 7000 according to various programs. The integrated control unit 7600 is connected to the input unit 7800. The input unit 7800 is implemented by a device such as a touch panel, a button, a microphone, a switch, a lever, etc. that can be input by the occupant. The integrated control unit 7600 can be supplied with data obtained by voice recognition of the voice input through the microphone. The input part 7800 can be, for example, a remote control device using infrared or other radio waves, or an external connection device that supports the operation of the vehicle control system 7000, such as a mobile phone, a personal digital assistant (PDA), etc. The input unit 7800 can be, for example, a camera. In this case, the occupant can input information through gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the occupant can be input. In addition, the input unit 7800 can include, for example, an input control circuit, etc., which generates an input signal based on the information input by the occupant, etc. using the above-mentioned input unit 7800, and outputs the generated input signal to the integrated control unit 7600. The occupants and the like input various data or instruct processing operations to the vehicle control system 7000 by operating the input unit 7800 .

[0222] The storage unit 7690 may include a read-only memory (ROM) storing various programs executed by the microcomputer and a random access memory (RAM) storing various parameters, operation results, sensor values, etc. In addition, the storage unit 7690 may be implemented by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc.

[0223] The general communication I / F 7620 is a widely used communication I / F that mediates communication with various devices existing in the external environment 7750. The general communication I / F 7620 may implement a cellular communication protocol such as Global System for Mobile Communications (GSM (registered trademark)), Worldwide Interoperability for Microwave Access (WiMAX (registered trademark)), Long Term Evolution (LTE (registered trademark)), LTE-Advanced (LTE-A), etc., or another wireless communication protocol such as a wireless LAN (also known as Wireless Fidelity (Wi-Fi (registered trademark)), Bluetooth (registered trademark), etc. For example, the general communication I / F 7620 may be connected to a device (e.g., an application server or a control server) existing on an external network (e.g., the Internet, a cloud network, or a company-specific network) via a base station or an access point. In addition, for example, the general communication I / F 7620 may be connected to a terminal existing near the vehicle (the terminal is, for example, a driver's terminal, a pedestrian or a store, or a machine type communication (MTC) terminal) using a peer-to-peer (P2P) technology.

[0224] The dedicated communication I / F 7630 is a communication I / F that supports the development of a communication protocol for use in a vehicle. The dedicated communication I / F 7630 may implement a standard protocol, such as Wireless Access in Vehicular Environments (WAVE) (which is a combination of the Institute of Electrical and Electronics Engineers (IEEE) 802.11p as a lower layer and IEEE 38609 as a higher layer), a dedicated short-range communication (DSRC), or a cellular communication protocol. The dedicated communication I / F 7630 generally performs V2X communication as a concept including one or more of the following: communication between vehicles and vehicles (vehicle to vehicle), communication between roads and vehicles (vehicle to infrastructure), communication between vehicles and homes (vehicle to home), and communication between pedestrians and vehicles (vehicle to pedestrians).

[0225] The positioning unit 7640 performs positioning, for example, by receiving a global navigation satellite system (GNSS) signal (e.g., a GPS signal from a global positioning system (GPS) satellite) from a GNSS satellite, and generates position information including the latitude, longitude, and altitude of the vehicle. Incidentally, the positioning unit 7640 may identify the current position by exchanging signals with a wireless access point, or may obtain position information from a terminal such as a mobile phone, a personal handyphone system (PHS), or a smart phone having a positioning function.

[0226] For example, the beacon receiving section 7650 receives radio waves or electromagnetic waves emitted from a radio station installed on a road or the like, and thereby obtains information on the current position, congestion, closed roads, necessary time, etc. Incidentally, the function of the beacon receiving section 7650 may be included in the above-mentioned dedicated communication I / F 7630.

[0227] The vehicle-mounted device I / F 7660 is a communication interface that mediates the connection between the microcomputer 7610 and various vehicle-mounted devices 7760 present in the vehicle. The vehicle-mounted device I / F 7660 can establish a wireless connection using a wireless communication protocol such as wireless LAN, Bluetooth (registered trademark), near field communication (NFC), or wireless universal serial bus (WUSB). In addition, the vehicle-mounted device I / F 7660 can establish a wired connection through a universal serial bus (USB), a high-definition multimedia interface (HDMI (registered trademark)), a mobile high-definition link (MHL), etc. via a connection terminal (and a cable, if necessary) not shown in the figure. The vehicle-mounted device 7760 may, for example, include at least one of a mobile device and a wearable device owned by an occupant and an information device carried or attached to the vehicle. The vehicle-mounted device 7760 may also include a navigation device that searches for a path to an arbitrary destination. The vehicle-mounted device I / F 7660 exchanges control signals or data signals with these vehicle-mounted devices 7760.

[0228] The in-vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals and the like according to a predetermined protocol supported by the communication network 7010.

[0229] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information obtained via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the vehicle-mounted device I / F 7660, and the vehicle-mounted network I / F 7680. For example, the microcomputer 7610 can calculate the control target value of the driving force generation device, the steering mechanism, or the braking device based on the information obtained about the inside and outside of the vehicle, and output a control command to the drive system control unit 7100. For example, the microcomputer 7610 can perform cooperative control aimed at realizing the functions of the advanced driver assistance system (ADAS), which includes collision prevention or shock absorption for the vehicle, follow-up driving based on the following distance, vehicle speed maintenance driving, warning of vehicle collision, warning of vehicle deviation from the lane, etc. In addition, microcomputer 7610 can perform cooperative control intended for autonomous driving by controlling a drive force generating device, a steering mechanism, a braking device, etc. based on information obtained about the vehicle's surroundings, thereby enabling the vehicle to travel automatically without depending on the driver's operation, etc.

[0230] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and objects such as surrounding structures, people, etc., based on information obtained via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the vehicle-mounted device I / F 7660, and the vehicle-mounted network I / F 7680, and generate local map information including information about the surroundings of the current position of the vehicle. In addition, the microcomputer 7610 may predict dangers such as collision of the vehicle, approach of pedestrians, etc., entering a closed road, etc., based on the obtained information, and generate a warning signal. The warning signal may be, for example, a signal for generating a warning sound or lighting a warning light.

[0231] The sound / image output portion 7670 transmits an output signal of at least one of sound and image to an output device that can visually or auditorily notify the occupants of the vehicle or the outside of the vehicle of information. Fig.38In the example of , an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are shown as output devices. For example, the display unit 7720 may include at least one of an onboard display and a head-up display. The display portion 7720 may have an augmented reality (AR) display function. The output device may be different from these devices, and may be another device such as headphones, a wearable device such as a glasses-type display worn by a passenger, a projector, a lamp, etc. In the case where the output device is a display device, the display device visually displays the results obtained by various processes performed by the microcomputer 7610 or information received from another control unit in various forms such as text, images, tables, graphics, etc. In addition, in the case where the output device is an audio output device, the audio output device converts an audio signal consisting of reproduced audio data or sound data, etc. into an analog signal, and outputs the analog signal audibly.

[0232] It should be noted that in Fig.38 In the example shown in , at least two control units connected via the communication network 7010 may be integrated into one control unit. Alternatively, each individual control unit may include a plurality of control units. In addition, the vehicle control system 7000 may include another control unit not shown in the figure. In addition, part or all of the functions performed by one of the control units described above may be assigned to another control unit. In other words, as long as information is sent and received via the communication network 7010, predetermined arithmetic processing may be performed by any control unit. Similarly, a sensor or device connected to one of the control units may be connected to another control unit, and a plurality of control units may send and receive detection information to each other via the communication network 7010.

[0233] It should be noted that the reference Figure 1 and Figure 2 The computer program for at least one function of the solid-state imaging device 1 according to the present embodiment described can be installed on any control unit or the like. In addition, a computer-readable storage medium in which such a computer program is stored can be provided. The storage medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above-mentioned computer program can be distributed on a network, for example, without using a storage medium.

[0234] In the above vehicle control system 7000, see Figure 1 The solid-state imaging device 1 according to the present embodiment described above can be applied to the vehicle exterior information detection unit 7400, the vehicle interior information detection unit 7500, or Fig.38Other appropriate components of the application examples shown. For example, by applying the solid-state imaging device 1 in the present disclosure to the vehicle exterior information detection unit 7400, it is possible to appropriately obtain information outside the vehicle, such as road conditions and conditions of other vehicles. In addition, by applying the solid-state imaging device 1 in the present disclosure to the vehicle-mounted information detection unit 7500, it is possible to appropriately obtain information inside the vehicle, such as the driver's condition or the passenger's condition.

[0235] In addition, you can Fig.38 The integrated control unit 7600 shown in FIG. 7 is implemented in a module (eg, an integrated circuit module including one die) as shown in FIG. Figure 1 Alternatively, reference may be made to at least a subset of the components of the solid-state imaging device 1 described herein. Figure 1 The solid-state imaging device 1 described can be Fig.38 The vehicle control system 7000 shown in FIG. 7 is implemented by multiple control units.

[0236] The above-mentioned embodiment can have the following modes. (1)

[0238] A solid-state imaging device comprising:

[0239] A plurality of pixels; and

[0240] control circuit, where

[0241] Each of the plurality of pixels includes:

[0242] A photoelectric conversion element performs photoelectric conversion on incident light to generate an electrical signal,

[0243] A first pixel circuit converts the electrical signal into first information; and

[0244] a second pixel circuit, converting the electrical signal into second information, and

[0245] Control Circuit

[0246] Each of the plurality of pixels is controlled so that the photoelectric conversion element is connected to the first pixel circuit or the second pixel circuit. (2)

[0248] The solid-state imaging device according to (1), wherein

[0249] The control circuit controls whether the first pixel circuit or the second pixel circuit converts the electric signal based on the arrangement of each pixel. (3)

[0251] The solid-state imaging device according to (2), wherein

[0252] The control circuit controls switching between output timing using the first pixel circuit and output timing using the second pixel circuit for each of the plurality of pixels. (4)

[0254] The solid-state imaging device according to (3), further comprising:

[0255] A pixel array, in which a plurality of pixels are arranged, wherein:

[0256] A plurality of pixels are arranged in a two-dimensional array along rows and columns extending in a direction intersecting the direction of the rows in the pixel array. (5)

[0258] The solid-state imaging device according to (4), wherein

[0259] The control circuit performs control for switching between pixels that perform output using the first pixel circuit and pixels that perform output using the second pixel circuit for each area in the pixel array. (6)

[0261] The solid-state imaging device according to (5), wherein

[0262] Photoelectric conversion elements are elements that obtain brightness information.

[0263] The first pixel circuit converts the electrical signal into grayscale information, and

[0264] The second pixel circuit converts the electrical signal into event detection information. (7)

[0266] The solid-state imaging device according to (5), wherein

[0267] The photoelectric conversion element is a single photon avalanche diode (SPAD).

[0268] The first pixel circuit converts the electrical signal into photon counting information, and

[0269] The second pixel circuit converts the electrical signal into information for acquiring time-of-flight (ToF) information. (8)

[0271] The solid-state imaging device according to any one of (5) to (7), wherein

[0272] The control circuit switches between the first pixel circuit and the second pixel circuit based on time. (9)

[0274] The solid-state imaging device according to (8), wherein

[0275] The control circuit switches between the first pixel circuit and the second pixel circuit at periodic timing. (10)

[0277] The solid-state imaging device according to (8) or (9), wherein:

[0278] The control circuit performs control to spatially periodically arrange pixels using the first pixel circuit and pixels using the second pixel circuit in the pixel array. (11)

[0280] The solid-state imaging device according to (10), wherein

[0281] The control circuit performs control to output the same type of signals, ie, the first signal or the second signal, from pixels belonging to the same row. (12)

[0283] The solid-state imaging device according to (11), wherein

[0284] The control circuit performs control to output different types of signals, ie, first signals or second signals, from pixels belonging to adjacent rows. (13)

[0286] The solid-state imaging device according to (11), wherein

[0287] The control circuit controls rows adjacent to the row outputting the first signal so that one of the rows is the row outputting the first signal and the other row is the row outputting the second signal, and controls the row outputting the second signal so that both adjacent rows are rows outputting the first signal. (14)

[0289] The solid-state imaging device according to (11), wherein

[0290] The control circuit divides the pixel array into regions of 2×2 pixels and performs control to arrange at least one pixel outputting the first signal and pixels outputting the second signal for pixels included in the range of each 2×2 pixels, instead of at least one pixel outputting the first signal. (15)

[0292] The solid-state imaging device according to (14), wherein

[0293] The control circuit performs control so that the pixel outputting the second signal moves periodically in the four-phase cycle. (16)

[0295] The solid-state imaging device according to (11), wherein

[0296] The control circuit divides the pixel array into regions of 3×3 pixels and performs control to arrange at least one pixel outputting the first signal and a pixel outputting the second signal for pixels included in the range of each 3×3 pixels, instead of at least one pixel outputting the first signal. (17)

[0298] The solid-state imaging device according to (16), wherein

[0299] The control circuit performs control so that the pixel outputting the second signal is periodically moved in the nine-phase cycle. (18)

[0301] The solid-state imaging device according to (6), wherein

[0302] Control Circuit

[0303] obtaining event information based on a second signal in the pixel array, and

[0304] According to the event information, a timing of selecting whether each of the plurality of pixels is connected to the first pixel circuit or the second pixel circuit is determined. (19)

[0306] The solid-state imaging device according to (18), wherein

[0307] Control Circuit

[0308] obtain event information in a region in the pixel array, and

[0309] According to the event information belonging to the area, it is selected whether the pixels belonging to the area are connected to the first pixel circuit or the second pixel circuit. (20)

[0311] The solid-state imaging device according to (18), wherein

[0312] Control Circuit

[0313] Obtain the optical flow of moving objects present in the pixel array,

[0314] Setting a region in the pixel array based on the optical flow, and

[0315] It is selected whether pixels belonging to a region are connected to a first pixel circuit or to a second pixel circuit. (twenty one)

[0317] The solid-state imaging device according to (6), wherein

[0318] Control Circuit

[0319] obtaining brightness information based on a first signal in the pixel array, and

[0320] The timing of selecting whether each of the plurality of pixels is connected to the first pixel circuit or the second pixel circuit is determined according to the brightness information. (twenty two)

[0322] The solid-state imaging device according to (21), wherein

[0323] Control Circuit

[0324] Based on the intensity of the brightness information, setting the region in the pixel array, and

[0325] It is selected whether pixels belonging to a region are connected to a first pixel circuit or to a second pixel circuit. (twenty three)

[0327] The solid-state imaging device according to (21), wherein

[0328] Control Circuit

[0329] Based on the result of object recognition using brightness information, a region is set in the pixel array, and

[0330] It is selected whether pixels belonging to a region are connected to a first pixel circuit or to a second pixel circuit.

[0331] The aspects of the present disclosure are not limited to the above-described embodiments, and include various conceivable variations. The effects of the present disclosure are not limited to the above-described contents. The components in each embodiment may be appropriately combined and applied. That is, various additions, modifications, and partial deletions may be made without departing from the conceptual concept and gist of the present disclosure obtained from the contents defined in the claims and their equivalents, etc.

[0332] Reference Symbols List

[0333] 1 Solid-state imaging device

[0334] 10 Solid-state imaging element

[0335] 100 pixel array

[0336] 102 Control Circuit

[0337] 104 Line drive circuit

[0338] 106 Column drive circuit

[0339] 108 signal processing circuit

[0340] 110 pixels

[0341] 112 Photoelectric conversion element

[0342] 114 Driving Circuit

[0343] 116 Switch

[0344] 118 first pixel circuit

[0345] 120 second pixel circuit

[0346] 16 Control Unit

[0347] 17 Storage Unit

[0348] 18 Interface

[0349] 19 Optical System

Claims

1. A solid-state imaging device, comprising: Multiple pixels; as well as control circuit, where Each of the plurality of pixels comprises: A photoelectric conversion element that performs photoelectric conversion on incident light to generate an electrical signal; A first pixel circuit converts the electrical signal into first information; and a second pixel circuit, converting the electrical signal into second information, and The control circuit Each of the plurality of pixels is controlled so that the photoelectric conversion element is connected to the first pixel circuit or the second pixel circuit.

2. The solid-state imaging device according to claim 1, wherein The control circuit controls the first pixel circuit or the second pixel circuit to convert the electric signal based on the arrangement of each of the pixels.

3. The solid-state imaging device according to claim 2, wherein: The control circuit controls switching between an output timing using the first pixel circuit and an output timing using the second pixel circuit for each of the plurality of pixels.

4. The solid-state imaging device according to claim 3, further comprising: A pixel array, wherein the plurality of pixels are arranged in the pixel array, wherein: The plurality of pixels are arranged in a two-dimensional array along rows in the pixel array and columns extending in a direction intersecting the direction of the rows.

5. The solid-state imaging device according to claim 4, wherein The control circuit performs control of switching between pixels that perform the output using the first pixel circuit and pixels that perform the output using the second pixel circuit for each area in the pixel array.

6. The solid-state imaging device according to claim 5, wherein The photoelectric conversion element is an element for obtaining brightness information. The first pixel circuit converts the electrical signal into grayscale information, and The second pixel circuit converts the electrical signal into event detection information.

7. The solid-state imaging device according to claim 5, wherein The photoelectric conversion element is a single photon avalanche diode SPAD, The first pixel circuit converts the electrical signal into photon counting information, and the second pixel circuit converts the electrical signal into information for obtaining time-of-flight ToF information.

8. The solid-state imaging device according to claim 5, wherein The control circuit switches between the first pixel circuit and the second pixel circuit based on time.

9. The solid-state imaging device according to claim 8, wherein The control circuit switches between the first pixel circuit and the second pixel circuit at periodic timing.

10. The solid-state imaging device according to claim 8, wherein The control circuit performs control to spatially periodically arrange pixels using the first pixel circuit and pixels using the second pixel circuit in the pixel array.

11. The solid-state imaging device according to claim 10, wherein The control circuit performs control to output the same type of signals, ie, the first signal or the second signal, from pixels belonging to the same row.

12. The solid-state imaging device according to claim 11, wherein The control circuit performs control of outputting different types of signals, ie, the first signal or the second signal, from pixels belonging to adjacent rows.

13. The solid-state imaging device according to claim 11, wherein The control circuit divides the pixel array into 2×2 pixel areas and performs control of arranging at least one pixel outputting the first signal and pixels outputting the second signal other than the at least one pixel outputting the first signal for the pixels included in the range of each 2×2 pixel.

14. The solid-state imaging device according to claim 11, wherein The control circuit divides the pixel array into regions of 3×3 pixels and performs control of arranging at least one pixel outputting the first signal and pixels outputting the second signal other than the at least one pixel outputting the first signal for the pixels included in the range of each 3×3 pixels.

15. The solid-state imaging device according to claim 6, wherein The control circuit obtaining event information based on a second signal in the pixel array, and The timing of selecting whether each of the plurality of pixels is connected to the first pixel circuit or the second pixel circuit is determined based on the event information.

16. The solid-state imaging device according to claim 15, wherein The control circuit obtaining the event information in the area in the pixel array, and Based on the event information belonging to the area, it is selected whether the pixels belonging to the area are connected to the first pixel circuit or the second pixel circuit.

17. The solid-state imaging device according to claim 15, wherein The control circuit obtaining the optical flow of a moving object present in the pixel array, setting a region in the pixel array based on the optical flow, and It is selected whether pixels belonging to the area are connected to the first pixel circuit or the second pixel circuit.

18. The solid-state imaging device according to claim 6, wherein The control circuit obtaining brightness information based on a first signal in the pixel array, and Based on the brightness information, a timing of selecting whether each of the plurality of pixels is connected to the first pixel circuit or the second pixel circuit is determined.

19. The solid-state imaging device according to claim 18, wherein The control circuit An area is set in the pixel array based on the intensity of the brightness information, and it is selected whether pixels belonging to the area are connected to the first pixel circuit or the second pixel circuit.

20. The solid-state imaging device according to claim 18, wherein The control circuit Based on a result of object recognition using the brightness information, a region is set in the pixel array, and It is selected whether pixels belonging to the area are connected to the first pixel circuit or the second pixel circuit.

Images