Light detection device and application processor

Abstract

The present invention optimizes the appropriate input / output characteristics of a light detection device and achieves an appropriate dynamic range with respect to incident light. In order to optimize appropriate input / output characteristics of a light detection device and achieve an appropriate dynamic range with respect to incident light, a light detection device of the present technology comprises: a light detection element that detects incidence of photons; an output unit that outputs an imaging result of the light detection element; a charging unit that charges the light detection element; and a control unit that controls the charging unit with respect to at least one of the number of detection intervals at which the photodetection element detects the photons, on the basis of the imaging result.

Description

TECHNICAL FIELD

[0001] The present technology relates to a light detection device. More particularly, the present technology relates to a light detection device for detecting photon incidence and an application processor for controlling the light detection device. BACKGROUND

[0002] Conventionally, a photoelectric conversion device is known that includes a pixel array unit in which pixels including avalanche photodiodes are arranged in a two-dimensional array in planar form. For example, a photoelectric conversion device has been proposed that changes a detection interval for detecting photon incidence depending on an exposure period (see, for example, Patent Literature 1). LIST OF CITATIONS PATENT LITERATURE

[0003] Patent Literature 1: JP 2022-106660 A SUMMARY TECHNICAL PROBLEM

[0004] According to the above-described related art, the power consumption of a photoelectric conversion device is suppressed by changing a detection interval depending on an exposure period. However, in the case where the exposure period is different, the above-described photoelectric conversion device controls only the number and period of a pulse signal within the exposure period, and there is a concern that some of the incident light can not fall within a suitable dynamic range.

[0005] The present technology is made in view of such a situation, and aims to achieve a suitable input / output characteristic of a light detection device and a suitable incident light dynamic range in a light detection device for detecting photon incidence. SOLUTION TO PROBLEM

[0006] The present technology is made in view of such a situation, and aims to achieve a suitable input / output characteristic of a light detection device and a suitable incident light dynamic range in a light detection device for detecting photon incidence.

[0007] In the first aspect, the light detecting device can include: a pixel array unit in which the light detecting elements are arranged in an array; a setting unit that sets a detection interval and a number of detections of the photons; an image processing unit that generates an image signal based on a signal of the pixel array unit output from the output unit as the imaging result; and a feature extraction unit that extracts a feature signal based on the imaging result, and the control unit can control setting information of the setting unit based on the feature signal extracted by the feature extraction unit. This results in an effect that proper input / output characteristics of a read circuit can be achieved to achieve a suitable dynamic range with respect to incident light.

[0008] In the first aspect, the feature signal can be a dynamic range based on histogram detection of a detection frequency of the photons represented as a frequency number, a maximum value or a minimum value of an abscissa of the histogram, an average luminance value, or a median of a maximum detection frequency. This results in an effect that proper input / output characteristics of a light detecting device can be achieved to achieve a suitable dynamic range with respect to incident light.

[0009] In the first aspect, the image processing unit can include a linearization circuit in an input stage, and the feature extraction unit can extract the feature signal based on a signal of the pixel array unit after passing through the linearization circuit. This results in an effect that proper input / output characteristics of a read circuit can be achieved to achieve a suitable dynamic range with respect to incident light.

[0010] In the first aspect, the pixel array unit can output a count value of the photons, and the image processing unit can include: a lookup table generation unit that generates a lookup table associating the count value with a photon rate based on the detection interval and the number of detections, and the linearization circuit that converts the count value to the photon rate with reference to the lookup table, and provides the photon rate to the feature extraction unit. This can achieve an effect that linearization is properly performed.

[0011] In the first aspect, the lookup table generation unit can generate the lookup table using an inverse function of a function representing a response characteristic of the pixel array unit in response to an input of the photons. This can achieve an effect that linearization is properly performed.

[0012] In the first aspect, the setting unit can set two or more detection intervals, and the lookup table generation unit can generate the lookup table by calculating the photon rate for each of the count values using the inverse function for each of the detection intervals, sorting a set of the photon rates calculated in a predetermined rule, and assigning the count value to each of the photon rates sorted. This achieves an effect that linearization can be properly performed when a plurality of detection intervals are set.

[0013] In the first aspect, the setting unit can set two or more detection intervals, and the look-up table can include a plurality of combination source look-up tables in which the detection intervals are different and a combination target look-up table, the look-up table generation unit can generate the combination target look-up table by weighted addition of the photon rates in each of the plurality of combination source look-up tables, and the linearization circuit can refer to the combination target look-up table. This has the effect of reducing the amount of calculation.

[0014] In the first aspect, the pixel array unit, the setting unit, and the output unit can be arranged on a first chip, and the control unit, the image processing unit, and the feature extraction unit can be arranged on a second chip. In this arrangement example, the read circuit can also achieve appropriate input / output characteristics, resulting in the effect that a suitable dynamic range with respect to incident light can be achieved.

[0015] In the first aspect, the pixel array unit, the setting unit, and the output unit can be arranged on the first chip, the image processing unit and the feature extraction unit can be arranged on the second chip, and the control unit can be arranged on a third chip. In this arrangement example, the read circuit can also achieve appropriate input / output characteristics, resulting in the effect that a suitable dynamic range with respect to incident light can be achieved.

[0016] In the first aspect, the pixel array unit, the setting unit, the output unit, and the control unit can be arranged on the first chip, and the image processing unit and the feature extraction unit can be arranged on the second chip. In this arrangement example, appropriate input / output characteristics of the read circuit can also be achieved, and also result in the effect that a suitable dynamic range with respect to incident light can be achieved.

[0017] In the first aspect, the feature extraction unit can extract the feature signal based on the image signal generated by the image processing unit. This results in the effect that appropriate input / output characteristics of the read circuit can be achieved, resulting in a suitable dynamic range with respect to incident light.

[0018] In the first aspect, the pixel array unit, the setting unit, and the output unit can be arranged on a first chip, the image processing unit can be arranged on a second chip, and the control unit and the feature extraction unit can be arranged on a third chip. In this arrangement example, appropriate input / output characteristics of the read circuit can also be achieved, and also result in the effect that a suitable dynamic range with respect to incident light can be achieved.

[0019] In the first aspect, the feature extraction unit can include a linearization circuit in an input stage, and extract the feature signal based on a signal of the pixel array unit after passing through the linearization circuit. This results in an effect that appropriate input / output characteristics of the read circuit can be achieved to realize a suitable dynamic range with respect to incident light.

[0020] In the first aspect, the pixel array unit, the setting unit, the output unit, the control unit, and the feature extraction unit can be arranged on a first chip, and the image processing unit can be arranged on a second chip. In this arrangement example, appropriate input / output characteristics of the read circuit can also be achieved, and an effect that a suitable dynamic range with respect to incident light can be achieved is also resulted.

[0021] The control unit controls the setting information of the setting unit based on external control by a user. This results in an effect that the user can arbitrarily control at least one of the detection interval and the number of detections of the photons.

[0022] In the first aspect, the control unit can set a plurality of detection intervals as the detection interval of the photons, and perform the detection of the photons with the plurality of detection intervals. This results in an effect that high dynamic range imaging in which the number of photon detections has a gradation, a wider luminance range, and imaging in which the gradation is improved in a specific luminance range can be achieved.

[0023] The control unit can perform control based on a histogram of a luminance distribution in an entire image of a scene to maximize the number of detections of the photons with respect to a gradation of the luminance distribution. This results in an effect that appropriate detection intervals of the photons and photon count values with respect to the luminance distribution of the scene can be controlled to obtain a maximum amount of information of the scene.

[0024] In the first aspect, the control unit can perform control based on a histogram of a luminance distribution in a region of a subject as a target to maximize the number of detections of the photons with respect to a gradation of the luminance distribution. This results in an effect that appropriate detection intervals of the photons and photon count values with respect to the luminance distribution of the target subject can be controlled to maximize information of the target subject.

[0025] In the first aspect, the control unit can perform control to minimize the number of detections of the photons in a range not lower than a minimum SNR required in a luminance range of a scene when drawing a signal-to-noise ratio (SNR) curve of the plurality of detection intervals. This results in an effect that power consumption of the light detection device can be suppressed.

[0026] In the first aspect, the control unit can perform control to shift the dynamic range only without changing the shape of the sensor response curve created based on the photon rate frequency distribution created by the feature extraction unit. This results in an effect that the dynamic range can be shifted only with the tone fixed.

[0027] In the first aspect, the feature extraction unit can include a wave detection circuit that creates a frequency distribution of a photon rate based on the imaging result, and a target sensor response curve design unit that creates a sensor response curve as a target based on the frequency distribution of the photon rate, the image processing unit, the wave detection circuit, the target sensor response curve design unit, and the control unit can constitute an application processor, and the application processor can output a signal for controlling at least one of a detection interval and a detection number of the photons to a sensor chip on which the pixel array unit is formed. This results in an effect that the application processor can control at least one of a detection interval and a detection number of the photons.

[0028] In the first aspect, the feature extraction unit can include a wave detection circuit that creates a frequency distribution of a photon rate based on the imaging result, and a target sensor response curve design unit that creates a sensor response curve as a target based on the frequency distribution of the photon rate, the image processing unit, the wave detection circuit, and the control unit can constitute an application processor, and the application processor can output a signal for controlling at least one of a detection interval and a detection number of the photons to a sensor chip on which the pixel array unit and the target sensor response curve design unit are formed. This results in an effect that the application processor can control at least one of a detection interval and a detection number of the photons.

[0029] Further, the second aspect of the present technology is an application processor that uses an imaging result output from a light detection device as input, the light detection device including a light detection element that detects incidence of a photon, an output unit that outputs an imaging result of the light detection element, a charging unit that charges the light detection element, and a control unit that controls the charging unit with respect to at least one of a detection interval and a detection number of the photon detected by the light detection element based on the imaging result, the application processor detects a feature signal in the imaging result, and outputs a signal for controlling at least one of a detection interval and a detection number of the photon detected by the light detection element based on the feature signal. This results in an effect that an application processor that can achieve appropriate input / output characteristics of a light detection device and achieve a suitable dynamic range with respect to incident light can be provided. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1is a block diagram showing a configuration example of a light detection device in the first embodiment of the present technology. Figure 2 is a graph showing an example of a histogram generated by a feature extraction unit in the first embodiment of the present technology. Figure 3 is a block diagram showing a configuration example of a reading circuit in the first embodiment of the present technology. Figure 4 is a circuit diagram showing a configuration example of a pixel circuit in the first embodiment of the present technology. Figure 5 is a timing chart showing an operation example of a light receiving unit in the first embodiment of the present technology. Figure 6 is a block diagram showing a configuration example of a light detection device in the second embodiment of the present technology. Figure 7 is a block diagram showing a configuration example of a light detection device in the third embodiment of the present technology. Figure 8 is a block diagram showing a configuration example of a light detection device in the fourth embodiment of the present technology. Figure 9 is a block diagram showing a configuration example of a light detection device in the fifth embodiment of the present technology. Figure 10 is a block diagram showing a configuration example of a light detection device in the sixth embodiment of the present technology. Figure 11 is a block diagram showing a configuration example of a light detection device in the seventh embodiment of the present technology. Figure 12 is a block diagram showing a configuration example of a light detection device in the eighth embodiment of the present technology. Figure 13 is a block diagram showing a configuration example of a light detection device in the ninth embodiment of the present technology. Figure 14 is an explanatory diagram regarding a light photon detection interval required for imaging of a high-low brightness scene. Figure 15 is an explanatory diagram regarding a combination of a long light photon detection interval and a short light photon detection interval. Figure 16 is an explanatory diagram regarding tone control per brightness level. Figure 17 is a block diagram showing a configuration example of a light detection device in the tenth embodiment of the present technology. Figure 18 is a diagram showing a control flow in a tone priority mode (1) in the light detection device in the tenth embodiment of the present technology. Figure 19 is a diagram showing a control flow in a tone priority mode (2) in the light detection device in the tenth embodiment of the present technology. Figure 20 is a diagram for explaining a low-power-consumption priority mode in the light detection device in the tenth embodiment of the present technology. Figure 21 is a block diagram showing a configuration example of the light detection device in the eleventh embodiment of the present technology. Figure 22 is a diagram for explaining shutter time control in the light detection device in the eleventh embodiment of the present technology. Figure 23 is a block diagram showing a configuration example of the light detection device in the twelfth embodiment of the present technology. Figure 24 is a block diagram showing a configuration example of the light detection device in the thirteenth embodiment of the present technology. Figure 25 is a diagram showing a lookup example in the thirteenth embodiment of the present technology. Figure 26 is a diagram showing a photon rate example per detection interval in the thirteenth embodiment of the present technology. Figure 27 is a diagram for explaining a method of generating a lookup table in the thirteenth embodiment of the present technology. Figure 28 is a diagram for explaining a method of generating a lookup table in the first modification of the thirteenth embodiment of the present technology. Figure 29 is a block diagram showing a schematic configuration example of a vehicle control system. Figure 30 is a diagram showing an installation position example of an imaging unit. DETAILED DESCRIPTION

[0031] Embodiments (hereinafter referred to as examples) for implementing the present technology will be described below. The description will be given in the following order. 1. First Embodiment (Example: Detect a feature signal based on a signal of a pixel array unit, and change a setting item of a pulse generator based on the feature signal) 2. Second Embodiment (Example: Detect a feature signal based on an image signal generated by an image processing unit, and change a setting item of a pulse generator based on the feature signal) 3. Third Embodiment (Example: Directly input a signal of a pixel array unit to a feature extraction unit to detect a feature signal, and change a setting item of a pulse generator based on the feature signal) 4. Fourth Embodiment (Example: Control a setting item of a pulse generator based on a user control from the outside in addition to the configuration of the first embodiment) 5. Fifth Embodiment (Arrangement example (1) of components in a light detection device) 6. Sixth Embodiment (Example (2) of Arrangement of Components in a Light Detection Device) 7. Seventh Embodiment (Example (3) of Arrangement of Components in a Light Detection Device) 8. Eighth Embodiment (Example (4) of Arrangement of Components in a Light Detection Device) 9. Ninth Embodiment (Example (5) of Arrangement of Components in a Light Detection Device) 10. Scheme for Performing Photon Detection in Multiple Detection Intervals 11. Tenth Embodiment (Example: Scheme for Performing Photon Detection in Multiple Detection Intervals, Prioritizing Gradation and Low Power Consumption) 12. Eleventh Embodiment (Example: Controlling Shutter Time by Scheme for Performing Photon Detection in Multiple Detection Intervals) 13. Twelfth Embodiment (Example: Signal Statistics Obtained by Wave Detection by Application Processor Output to Sensor Chip Side, and Performing Processing at and After Design of Response Curve on Sensor Chip Side) 14. Thirteenth Embodiment (Example: Generating Lookup Table Using Inverse Function) 15. Modified Example 16. Application Example on Mobile Body 17. Configuration Adoptable by the Present Technology

[0032] <1. First Embodiment> [Configuration Example of Light Detection Device] Figure 1 is a block diagram showing a configuration example of a light detection device 100 in the first embodiment of the present technology. The light detection device 100 in the first embodiment includes a pulse generator 110, a pixel array unit 120, an interface circuit 130, a signal processing unit 140, and a control unit 150.

[0033] The pulse generator 110 generates various pulse signals for driving the pixel array unit 120 under the control of the control unit 150, and sets a detection interval and a detection number of photons. The pixel array unit 120 is an imaging unit that includes a plurality of read circuits 200 that perform photoelectric conversion and capture an image based on various pulse signals given from the pulse generator 110. The plurality of read circuits 200 are arranged two-dimensionally in a matrix (array), and output a signal according to the detection number of incident photons. The interface circuit 130 supplies a signal of the pixel array unit 120 to the signal processing unit 140.

[0034] Note that the signal of the pixel array unit 120 is an example of the imaging result described in the claims, the pixel array unit 120 and the interface circuit 130 are examples of the output unit described in the claims, and the pulse generator 110 is an example of the setting unit described in the claims.

[0035] The signal processing unit 140 is configured to include an image processing unit 160 and a feature extraction unit 170.

[0036] The image processing unit 160 includes a linearization circuit 161 provided in an input stage and an image signal generation unit 162 provided in a post stage. The linearization circuit 161 converts (linearizes) the input signal of the pixel array unit 120 into a linear signal proportional to the brightness. In the linearization process, the number of detections (counts) of photons is converted into an estimated number of incident photons. The image signal generation unit 162 generates an image signal based on the signal of the pixel array unit 120 linearized by the linearization circuit 161 and outputs the image signal to the outside of the signal processing unit 140.

[0037] The feature extraction unit 170 includes a wave detection circuit 180. The wave detection circuit 180 receives, as an input, the signal of the pixel array unit 120 linearized by the linearization circuit 161 in the image processing unit 160 and generates a histogram representing the frequency of photon detection as a frequency number based on the signal. Figure 2 An example of the histogram is shown. In the histogram shown in FIG. 6, the horizontal axis represents the brightness or the frequency of incident photons (the number of photons within the exposure period), and the vertical axis represents the number of pixels. Figure 2 In the histogram shown, the horizontal axis represents the brightness or the frequency of incident photons (the number of photons within the exposure period), and the vertical axis represents the number of pixels.

[0038] Based on the histogram generated by the wave detection circuit 180, the feature extraction unit 170 extracts a feature signal of the output value of the light detection device 100 to determine the feature of the image. Specifically, based on the histogram representing the frequency of photon detection as a frequency number, the feature extraction unit 170 extracts the dynamic range, the maximum value or the minimum value of the horizontal axis of the histogram, the average brightness value, or the median as a feature signal and provides the extracted feature signal to the control unit 150.

[0039] In the histogram shown in FIG. 6, the dynamic range is the difference between the minimum value and the maximum value of the horizontal axis, i.e., the difference between the minimum number of photons and the maximum number of photons. The average brightness value is the average value of all detection frequencies. The median is the median of the maximum detection frequency. The same applies to each of the embodiments described below for the feature signals such as the dynamic range, the maximum value and the minimum value of the horizontal axis of the histogram, the average brightness value, and the median. Figure 2

[0040] ​Based on the feature signal provided from the feature extraction unit 170, the control unit 150 performs control to change the setting information of the pulse generator 110. Specifically, based on the feature signal, the control unit 150 performs control to change at least one of the detection interval and the number of detections of the photons in the read circuit 200, preferably both the detection interval and the number of detections, as the setting information of the pulse generator 110. The control unit 150 also performs control on the image processing unit 160.

[0041] [Configuration example of read circuit] Figure 3 is a block diagram illustrating a configuration example of the read circuit 200 in the first embodiment of the present technology. The read circuit 200 in the first embodiment of the present technology includes a pixel circuit 210, a counter 220, and a selection switch 230.

[0042] The pixel circuit 210 generates a pulse signal PLS in response to the incidence of photons and provides the pulse signal PLS to the counter 220. The counter 220 counts the number of pulses of the pulse signal PLS and provides the count value as pixel data to the selection switch 230. The count value of the counter 220 is initialized by a reset signal RST provided from the pulse generator 110 illustrated in Figure 1

[0043] The selection switch 230 performs on / off operation in accordance with a selection signal SEL provided from the pulse generator 110 illustrated in Figure 1 Figure 1 and provides the count value of the counter 220 as pixel data to the signal processing unit 140 illustrated in

[0044] [Configuration example of pixel circuit] Figure 4 is a circuit diagram illustrating a configuration example of the pixel circuit 210 in the first embodiment of the present technology. The pixel circuit 210 in the first embodiment of the present technology includes a light detecting element 211, a clamp transistor 212, an inverter 213, a charging transistor 214, and a detection circuit 240. For example, the clamp transistor 212 and the charging transistor 214 can use a p-channel metal oxide semiconductor (pMOS) transistor. Note that the charging transistor 214 is an example of the charging unit described in the claims.

[0045] In the pixel circuit 210, the light detecting element 211 among the light detecting element 211, the clamp transistor 212, the inverter 213, and the charging transistor 214 is arranged in an array form on a substrate different from a substrate of the detection circuit 240. Note that some of the clamp transistor 212, the inverter 213, and the charging transistor 214 can be arranged on the same substrate as the substrate of the light detecting element 211.​​

[0046] The light-detecting element 211 can use an avalanche photodiode (APD). In a Geiger mode avalanche photodiode, if a voltage equal to or greater than a breakdown voltage is applied between terminals, an avalanche phenomenon occurs due to the incidence of a single photon. An avalanche photodiode that multiplies a single photon by the avalanche phenomenon is called a single photon avalanche diode (SPAD). Here, an exemplary case where a SPAD element is used as the light-detecting element 211 is explained. In other words, the SPAD element is an example of the light-detecting element described in the claims.

[0047] The anode of each light-detecting element 211 is connected to a node having a predetermined potential lower than the power supply voltage VDD, and its cathode is connected to the clamp transistor 212. The cathode potential of the light-detecting element 211 is defined as Vk1. DD

[0048] The clamp transistor 212 and the charge transistor 214 are connected in series between the node of the power supply voltage VDD and the cathode of the light-detecting element 211, with the charge transistor 214 on the power supply voltage VDD side. The potential of the connection node (hereinafter referred to as the detection node N1) between the clamp transistor 212 and the charge transistor 214 is defined as Vk2.

[0049] The control signal CLIP provided from the pulse generator 110 is input to the gate of the clamp transistor 212. Also, the control signal XRST provided from the pulse generator 110 is input to the gate of the charge transistor 214 and the inverter 213.

[0050] Note that, Figure 4 A read scheme in which the charge transistor 214 is connected to the cathode of the light-detecting element 211 is shown, but a read scheme in which the charge transistor 214 is connected to the anode of the light-detecting element 211 and the node of the power supply voltage VDD is connected to the cathode can also be adopted.

[0051] The inverter 213 inverts the logic of the control signal XRST and provides it to the detection circuit 240.

[0052] The detection circuit 240 detects the incidence of a photon and generates a pulse signal PLS. The detection circuit 240 includes a pMOS transistor 241, an n-channel MOS transistor (nMOS) 242, an inverter 243, and an inverter 244.

[0053] ​The pMOS transistor 241 and the nMOS transistor 242 are connected in series between a node of the power supply voltage VDD and a node of the ground voltage, with the pMOS transistor 241 being on the power supply voltage VDD side. The gate of the pMOS transistor 241 is connected to the detection node N1. The inverted signal of the control signal XRST from the inverter 213 is input to the gate of the nMOS transistor 242. A signal corresponding to the potential of the connection node N2 between the pMOS transistor 241 and the nMOS transistor 242 is defined as the detection signal DET.

[0054] The inverter 243 is configured to invert the logic of the detection signal DET. The inverter 244 inverts the logic of the inverted signal from the inverter 243 and supplies it as the pulse signal PLS to Figure 3 the counter 220 shown.

[0055] Further, setting information for controlling the reading circuit 200 is input from the control unit 150 to the pulse generator 110. The setting information includes a set value of at least one of a detection interval at which a photon is detected to be incident and a number of detections. Here, the detection interval indicates a period in which a photon is detected to be incident, and the number of detections indicates the number of times of control of repeating the detection interval. Since photon detection is performed within an exposure period, the detection interval x the number of detections is the same as the value of the exposure period.

[0056] Within the detection interval, the control unit 150 performs control on the pulse generator 110 to set the control signal CLIP to the high level and then to the low level. Within the period in which the control signal CLIP is at the high level within the detection interval, the detection node N1 is disconnected from the cathode of the light detection element 211. Hereinafter, this period is referred to as a "standby period" in which a photon is awaited to be incident. On the other hand, within the period in which the control signal CLIP is at the low level within the detection interval, the detection node N1 is connected to the cathode of the light detection element 211. Hereinafter, this period is referred to as a "connection period".

[0057] Once a photon is incident during the standby period, avalanche multiplication occurs in the light detection element 211, and its cathode potential Vk1 falls to a certain potential. The clamping transistor 212 connects the detection node N1 to the light detection element 211 during the subsequent connection period. Therefore, in the case where a photon is incident during the standby period, charge is transferred to the detection node N1 during the connection period, and the pulse signal PLS is set to the high level.

[0058] Further, the control unit 150 performs control on the pulse generator 110 to set the control signal XRST to a low level for a predetermined period from a predetermined charging start time in the connection period. In this way, the charging transistor 214 supplies the power supply voltage VDD to the detection node N1. Since the detection node N1 is connected to the cathode of the light detection element 211, a charging operation is performed to return the cathode potential Vk1 to the power supply voltage VDD. Meanwhile, the detection circuit 240 is initialized by the low-level control signal XRST.

[0059] In short, the clamp transistor 212 electrically connects the light detection element 211 to the detection node N1 in the connection period, and disconnects the light detection element 211 from the detection N1 in the standby period. Further, in the connection period, the charging transistor 214 supplies the power supply voltage VDD to the detection node N1 after the charging start time until a predetermined period elapses.

[0060] [Operation example of light receiving unit] Figure 5 is a timing chart showing an operation example of the pixel circuit 210 in the first embodiment of the present technology. Figure 5 The timing relationship between the control signal CLIP, the control signal XRST, the cathode potential Vk1, the potential Vk2 of the detection node N1, and the output signal of the inverter 213 is shown.

[0061] In the initial state, both the cathode potential Vk1 and the potential Vk3 of the detection node are the power supply voltage VDD.

[0062] In the detection interval from time T0 to time T3, the period until time T1 corresponds to the standby period, and the period from time T1 to time T3 corresponds to the connection period. In the standby period, the pulse generator 110 sets the control signal CLIP to a high level under the control of the control unit 150. Further, in the standby period, the pulse generator 110 sets the control signal XRST to a high level for a certain period from a predetermined time.

[0063] It is assumed that a photon is incident during the standby period from time T0 to time T1. In the drawing, a thick arrow indicates the time of incidence of the photon. Due to the incidence of the photon, avalanche multiplication occurs in the light detection element 211 (SPAD element), the cathode potential Vk1 drops to a certain potential, and the avalanche multiplication stops. Since the detection node N1 is disconnected from the light detection element 211 during the standby period, the potential Vk2 of the detection node N1 does not fluctuate.

[0064] When the cathode of the light-detecting element 211 is connected to the detection node N1 at time T1, part of the charge of the cathode moves to the detection node N1. In this way, the potential Vk2 of the detection node N1 drops, and the cathode potential Vk1 rises by an amount corresponding to the amount of charge moved. When the potential Vk2 of the detection node N1 becomes equal to or less than the threshold value of the pMOS transistor 241, the pMOS transistor 241 enters the on state, and the pulse signal PLS rises.

[0065] Under the control of the control unit 150, the pulse generator 110 sets the control signal XRST to the low level for a predetermined period from time T2 within the connection period. In this way, the power supply voltage VDD is supplied, and the potential Vk2 of the detection node N1 is initialized to the power supply voltage VDD. Since the detection node N1 is connected to the cathode of the light-detecting element 211, the cathode potential Vk1 is also initialized to the power supply voltage VDD. At the same time, the detection circuit 240 is initialized, and the pulse signal PLS drops.

[0066] As described above, the pixel circuit 210 in the first embodiment of the present technology performs control so that the read circuit 200 enters the detection state of detecting photons during the period from time T1 to time T2 within the connection period, and the read circuit 200 enters the charge state during the period from time T2 to time T3.

[0067] By repeating the above-described control within the detection interval, one pulse can be generated within each detection interval when a photon is incident one or more times during the standby period. For example, since there is no photon incident in the detection interval from time T3 to time T4 within the standby period, no pulse is generated during the connection period. On the other hand, since there are two photon incidents within the detection interval from time T4 to time T5, one pulse is generated.

[0068] For the above-described configuration example of the read circuit 200, the configuration example of the pixel circuit 210, and the operation example of the pixel circuit 210 in the first embodiment of the present technology, the same applies to each of the embodiments described below.

[0069] As described above, the light detection device 100 in the first embodiment of the present technology includes the read circuit 200 that includes the pixel circuit 210 that includes the light detection element 211 and the charge transistor 214 for charging the light detection element 211, and that performs imaging by counting whether or not a photon is incident within a defined detection interval. Then, at least one of the detection interval and the number of detections of the photon is controlled based on the imaging result. More specifically, a characteristic signal is detected based on the signal of the pixel array unit 120 that has been converted into a linear signal proportional to luminance by the linearization circuit 161 in the image processing unit 160, and at least one of the detection interval and the number of detections of the photon is controlled based on the characteristic signal. In this way, appropriate input / output characteristics (i.e., characteristics of the number of incident photons - the count value of the counter 220) of the read circuit 200 can be achieved, and a suitable dynamic range with respect to incident light can also be achieved.

[0070] Further, in the first embodiment of the present technology, by the effect of the connection / disconnection (cut-off) between the cathode of the light detection element 211 (SPAD element) and the detection node N1 by the clamp transistor 212 in the light detection device 100, the capacitance of the cathode of the light detection element 211 can be reduced. Therefore, power consumption can be reduced.

[0071] <2. Second Embodiment> [Configuration Example of Light Detection Device] Figure 6 is a block diagram that shows a configuration example of the light detection device 100 in the second embodiment of the present technology.

[0072] The light detection device 100 in the first embodiment of the present technology is configured so that the linearization circuit 161 of the image processing unit 160 inputs the signal of the pixel array unit 120 that has been converted into a linear signal proportional to luminance to the feature extraction unit 170. On the other hand, the light detection device 100 in the second embodiment of the present technology is configured to input an image signal generated by the image signal generation unit 162 in the image processing unit 160 to the feature extraction unit 170.

[0073] The image processing unit 160 includes the linearization circuit 161 and the image signal generation unit 162 that generates an image signal based on the signal of the pixel array unit 120 that has been converted into a linear signal proportional to luminance by the linearization circuit 161, and the image processing unit 160 outputs the image signal to the outside of the signal processing unit 140.

[0074] In the feature extraction unit 170, the wave detection circuit 180 receives the image signal generated by the image signal generation unit 162 in the image processing unit 160 as input, and generates a histogram in which the photon detection frequency is expressed as the frequency number on the basis of the pixel signal. Then, the feature extraction unit 170 extracts the dynamic range, the average luminance value, or the median as the feature signal on the basis of the histogram generated by the wave detection circuit 180, and supplies the extracted feature signal to the control unit 150.

[0075] On the basis of the feature signal supplied from the feature extraction unit 170, the control unit 150 performs control to change at least one of the detection interval and the number of detections of photons in the read circuit 200, preferably both the detection interval and the number of detections, as the setting information of the pulse generator 110.

[0076] As described above, the light detection device 100 in the second embodiment of the present technology detects a feature signal on the basis of the image signal generated by the image processing unit 160, and controls at least one of the detection interval and the number of detections of photons on the basis of the feature signal. In this way, appropriate input / output characteristics of the read circuit 200 can be achieved, and similarly to the first embodiment in which a feature signal is detected on the basis of a signal of the pixel array unit 120 that has been converted to a linear signal proportional to the luminance by the linearization circuit 161 in the image processing unit 160, appropriate dynamic range with respect to incident light can be achieved.

[0077] <3. Third Embodiment> [Configuration Example of Light Detection Device] Figure 7 is a block diagram showing a configuration example of the light detection device 100 in the third embodiment of the present technology.

[0078] The light detection device 100 in the third embodiment of the present technology is configured so that the signal of the pixel array unit 120 output from the interface circuit 130 is directly input to the feature extraction unit 170. Therefore, the feature extraction unit 170 is configured to include the linearization circuit 190 located in the front stage (i.e., the input stage) of the wave detection circuit 180. Similarly to the linearization circuit 161 provided in the image processing unit 160 in the light detection device 100 in the first embodiment of the present technology, the linearization circuit 190 linearizes the input signal of the pixel array unit 120 to a linear signal proportional to the luminance.

[0079] In the feature extraction unit 170, the wave detection circuit 180 generates a histogram representing the frequency of photon detection as a frequency number based on the signal of the pixel array unit 120 that has been converted to a linear signal proportional to luminance by the linearization circuit 190. Then, the feature extraction unit 170 extracts a feature signal of the output value of the light detection device 100 based on the histogram generated by the wave detection circuit 180, and supplies the extracted feature signal to the control unit 150.

[0080] The control unit 150 performs control to change at least one of the detection interval and the number of detections of photons in the read circuit 200, preferably both the detection interval and the number of detections, as the setting information of the pulse generator 110, based on the feature signal supplied from the feature extraction unit 170.

[0081] As described above, the light detection device 100 in the third embodiment of the present technology includes the linearization circuit 190 at the input stage of the feature extraction unit 170, and the signal of the pixel array unit 120 output from the interface circuit 130 is directly input to the feature extraction unit 170. Further, by detecting a feature signal based on the signal of the pixel array unit 120 that has been linearized to a linear signal proportional to luminance in the feature extraction unit 170, and controlling at least one of the detection interval and the number of detections of photons based on the feature signal, appropriate input / output characteristics of the read circuit 200 can be achieved, and a suitable dynamic range with respect to incident light similar to the first embodiment can be achieved.

[0082] <4. Fourth Embodiment> [Configuration Example of Light Detection Device] Figure 8 is a block diagram showing a configuration example of the light detection device 100 in the fourth embodiment of the present technology.

[0083] The light detection device 100 in the fourth embodiment of the present technology basically adopts the configuration of the light detection device 100 in the first embodiment of the present technology, and is also configured to control at least one of the detection interval and the number of detections of photons by the control unit 150 changing the setting information of the pulse generator 110 based on the feature signal detected by the feature extraction unit 170. In addition to this configuration, the light detection device 100 in the fourth embodiment of the present technology is configured so that the control unit 150 controls at least one of the detection interval and the number of detections of photons based on user control from the outside.

[0084] The external control (user control) of the control unit 150 by the user can be performed by a bus such as an internal integrated circuit (I2C), an improved internal integrated circuit (I3C), or a serial peripheral interface (SPI).

[0085] According to the light detection device 100 in the fourth embodiment of the present technology described above, the setting information of the pulse generator 110 can be changed in accordance with the characteristic signal detected based on the imaging result, and the setting information of the pulse generator 110 can be arbitrarily changed based on the external control performed by the user.

[0086] [Modified example of the fourth embodiment] As a modified example of the fourth embodiment, a system configuration can also be employed that can switch the following modes: a mode in which manual control is performed based on user control, a mode in which automatic control is performed based on an imaging result, and a mode in which control is performed based on a limit set by a user and an imaging result. As a target of the limit set by the user, examples can be a dynamic range, a shutter time, a frame rate, a signal-to-noise ratio (SNR), and a gamma curve characteristic.

[0087] <5. Fifth embodiment> [Arrangement example (1) of components in light detection device] Figure 9 is a block diagram illustrating an arrangement example of a light detection device 100 in a fifth embodiment of the present technology. The arrangement example of the light detection device 100 in the fifth embodiment of the present technology employs the configuration of the light detection device 100 in the first embodiment of the present technology.

[0088] The arrangement example of the light detection device 100 in the fifth embodiment of the present technology is configured such that the pulse generator 110, the pixel array unit 120, and the interface circuit 130 are arranged on a sensor chip 300, and the control unit 150, the image processing unit 160, and the feature extraction unit 170 are arranged on an image signal processor (ISP) chip 400 of the light detection device 100. Note that the sensor chip 300 is an example of the first chip described in the claims, and the ISP chip 400 is an example of the second chip described in the claims.

[0089] In this way, even in the arrangement example (1) in which the image processing unit 160 and the feature extraction unit 170 are arranged on the ISP chip 400, similar effects and advantages to the light detection device 100 in the first embodiment of the present technology can be obtained. In other words, at least one of the detection interval and the number of detections of photons can be controlled in accordance with the characteristic signal detected based on the imaging result, thereby achieving appropriate input / output characteristics of the read circuit 200 and achieving a suitable dynamic range with respect to incident light.

[0090] <6. Sixth embodiment> [Arrangement example (2) of components in light detection device] Figure 10is a block diagram illustrating an arrangement example of the light detection device 100 in the sixth embodiment of the present technology. The arrangement example of the light detection device 100 in the sixth embodiment of the present technology adopts the configuration of the light detection device 100 in the second embodiment of the present technology.

[0091] The arrangement example of the light detection device 100 in the sixth embodiment of the present technology is configured such that, in the light detection device 100 in the second embodiment of the present technology, the pulse generator 110, the pixel array unit 120, and the interface circuit 130 are arranged on the sensor chip 300, the image processing unit 160 is arranged on the ISP chip 400, and the control unit 150 and the feature extraction unit 170 are arranged on the control chip 500 different from the ISP chip 400. Note that the control chip 500 is an example of the third chip described in the claims.

[0092] In this way, even in the arrangement example (2) in which the control unit 150 and the feature extraction unit 170 are arranged on the control chip 500 that is a chip different from the ISP chip 400, similar effects and advantages to those of the light detection device 100 in the first embodiment of the present technology can be obtained. In other words, at least one of the detection interval and the number of detections of photons can be controlled in accordance with a feature signal detected based on an imaging result, thereby achieving appropriate input / output characteristics of the read circuit 200 and achieving a suitable dynamic range with respect to incident light.

[0093] <7. Seventh Embodiment> [Arrangement Example (3) of Components in Light Detection Device] Figure 11 is a block diagram illustrating an arrangement example of the light detection device 100 in the seventh embodiment of the present technology. The arrangement example of the light detection device 100 in the seventh embodiment of the present technology adopts the configuration of the light detection device 100 in the third embodiment of the present technology.

[0094] The arrangement example of the light detection device 100 in the seventh embodiment of the present technology is configured such that, in the light detection device 100 in the third embodiment of the present technology, the control unit 150 and the feature extraction unit 170 are arranged on the sensor chip 300 together with the pulse generator 110, the pixel array unit 120, and the interface circuit 130, while the image processing unit 160 is arranged on the ISP chip 400.

[0095] In this way, even in the arrangement example (3) in which the control unit 150 and the feature extraction unit 170 are arranged on the sensor chip 300 together with the pixel array unit 120, similar effects and advantages to those of the light detection apparatus 100 in the first embodiment of the present technology can be obtained. In other words, at least one of the detection interval and the number of detections of photons can be controlled in accordance with the feature signal detected based on the imaging result, thereby achieving appropriate input / output characteristics of the read circuit 200 and achieving a suitable dynamic range with respect to incident light.

[0096] <8. Eighth Embodiment> [Arrangement Example (4) of Components in Light Detection Apparatus] Figure 12 is a block diagram illustrating an arrangement example of the light detection apparatus 100 in the eighth embodiment of the present technology. The arrangement example of the light detection apparatus 100 in the eighth embodiment of the present technology adopts the configuration of the light detection apparatus 100 in the first embodiment of the present technology.

[0097] The arrangement example of the light detection apparatus 100 in the eighth embodiment of the present technology is configured such that, in the light detection apparatus 100 in the first embodiment of the present technology, the pulse generator 110, the pixel array unit 120, and the interface circuit 130 are arranged on the sensor chip 300, the image processing unit 160 and the feature extraction unit 170 are arranged on the ISP chip 400, and only the control unit 150 is arranged on the control chip 500.

[0098] In this way, even in the arrangement example (4) in which the control unit 150 is arranged on the control chip 500 different from the sensor chip 300 and the ISP chip 400, similar effects and advantages to those of the light detection apparatus 100 in the first embodiment of the present technology can be obtained. In other words, at least one of the detection interval and the number of detections of photons can be controlled in accordance with the feature signal detected based on the imaging result, thereby achieving appropriate input / output characteristics of the read circuit 200 and achieving a suitable dynamic range with respect to incident light.

[0099] <9. Ninth Embodiment> [Arrangement Example (5) of Components in Light Detection Apparatus] Figure 13 is a block diagram illustrating an arrangement example of the light detection apparatus 100 in the ninth embodiment of the present technology. The arrangement example of the light detection apparatus 100 in the ninth embodiment of the present technology adopts the configuration of the light detection apparatus 100 in the first embodiment of the present technology.

[0100] The arrangement example of the light detecting device 100 in the ninth embodiment is configured so that, in the light detecting device 100 in the first embodiment of the present technology, the control unit 150 is arranged on the sensor chip 300 together with the pulse generator 110, the pixel array unit 120, and the interface circuit 130, while the image processing unit 160 and the feature extraction unit 170 are arranged on the ISP chip 400.

[0101] In this way, even in the arrangement example (5) in which the control unit 150 is arranged on the sensor chip 300 together with the pixel array unit 120, similar effects and advantages to the light detecting device 100 in the first embodiment of the present technology can be obtained. In other words, at least one of the detection interval and the number of detections of photons can be controlled in accordance with the feature signal detected based on the imaging result, thereby realizing appropriate input / output characteristics of the read circuit 200 and realizing a suitable dynamic range with respect to incident light.

[0102] <10. Regarding a scheme of performing photon detection in multiple detection intervals> In a pixel circuit that detects a rise caused by avalanche multiplication and performs reset at a specific interval by a control signal provided from the outside, the luminance range in which the count value of the counter is allowed to have gradation is limited by the photon detection interval. On the other hand, by adopting a scheme of detecting photons (photon counting) in multiple detection intervals, high dynamic range imaging in a wider luminance range in which the number of detections of photons (the count value of the counter) is allowed to have gradation and imaging in which gradation is improved in a specific luminance range can be realized. Here, the luminance range is the dynamic range of luminance, and is the range from the darkest to the brightest in the entire image.

[0103] [Short detection interval at high luminance and long detection interval at low luminance] Hereinafter, a case in which two detection intervals (i.e., a relatively long detection interval and a relatively short detection interval) are taken as an example of multiple detection intervals will be described. Figure 14 The scheme of detecting photons in multiple detection intervals will be specifically described. Here, a case in which two detection intervals (i.e., a relatively long detection interval and a relatively short detection interval) are taken as an example of multiple detection intervals will be described. Figure 14 is a diagram of the photon detection interval required for imaging high-luminance and low-luminance scenes.

[0104] The relatively long photon detection interval and the relatively short photon detection interval constitute one imaging frame. Within one detection interval, if one or more photons are incident, the count value (count number) is incremented to "1". Even if a plurality of photons are incident within one detection interval, the count value is "1".

[0105] When imaging a scene with relatively low luminance (low luminance), the frequency of photon incidence is low. Therefore, by using a relatively long photon detection interval at low luminance, loss of photon counts can be prevented. In contrast, when imaging a scene with relatively high luminance (high luminance), the frequency of photon incidence is high, and if a relatively long photon detection interval is used, the count value will saturate. Therefore, since counting of photons can be performed while sparsification is performed using a relatively short photon detection interval at high luminance, saturation of the count value can be prevented.

[0106] As can be seen from the above description, for imaging of a scene with relatively high luminance, a relatively short photon detection interval is required, and for imaging of a scene with relatively low luminance, a relatively long photon detection interval is required.

[0107] Figure 14 The lower part of FIG. 1 illustrates input / output characteristics, in which the horizontal axis represents the incident photon rate proportional to luminance, and the vertical axis represents the count value. Here, the case of a relatively long photon detection interval (e.g., 1000 [nsec] x 5000 [cnt]) is represented by a solid line, and the case of a relatively short photon detection interval (e.g., 10 [nsec] x 5000 [cnt]) is represented by a broken line.

[0108] In the input / output characteristics, the portion in which the curve gradation exists corresponds to the dynamic range. As can be seen from the input / output characteristics, if only a relatively long detection interval or a relatively short detection interval is used, the dynamic range can be ensured only in a portion with relatively low luminance, or only in a portion with relatively high luminance.

[0109] [Combination of long and short detection intervals] Therefore, a relatively long photon detection interval and a relatively short photon detection interval are used in combination. By combining long and short detection intervals in this way, gradation can be exhibited over a wide luminance range, and thus a high dynamic range can be achieved.

[0110] Figure 15 FIG. 2 is an explanatory diagram of long and short photon detection interval combination. Figure 15 The lower part of FIG. 2 illustrates input / output characteristics, in which the horizontal axis represents the incident photon rate, and the vertical axis represents the count value. Here, the case of a relatively long photon detection interval (e.g., 1000 [nsec] x 5000 [cnt]) is represented by a solid line, and the case of a relatively short photon detection interval (e.g., 10 [nsec] x 5000 [cnt]) is represented by a dotted line.

[0111] In addition, the characteristics in the case of the combination of long and short detection intervals are indicated by broken line curves. The characteristics achieved by the combination of long and short detection intervals are, for example, the characteristics in the case of 10 [nsec] x 1666 [cnt] + 100 [nsec] x 1666 [cnt] + 1000 [nsec] x 1666 [cnt] and 10 [nsec] : 100 [nsec] : 1000 [nsec] = 1 : 1 : 1.

[0112] From Figure 15 As can be seen from the input / output characteristics in FIG. 6, by using a relatively long detection interval and a relatively short interval in combination, it is possible to appropriately detect (count) photons for any luminance, it is possible to exhibit a gradation in a wide luminance range, and it is possible to achieve a high dynamic range.

[0113] [Gradation control for each luminance level] In addition to the combination of long and short detection intervals, it is also possible to change the mixing ratio of the number of detections (count values) of each of the long and short detection intervals. By changing the mixing ratio of the number of detections of each of the long and short detection intervals in this way, it is possible to control the gradation at each luminance level.

[0114] Figure 16 FIG. 6 is a diagram of gradation control for each luminance level. Figure 16 Input / output characteristics are shown in which the horizontal axis indicates the incident photon rate and the vertical axis indicates the count value. In the input / output characteristics, the case of a relatively long detection interval is indicated by a solid line curve, the case of a relatively short detection interval is indicated by a dotted line curve, and the case of a combination of long and short detection intervals is indicated by a broken line curve.

[0115] Here, as an example, a longer detection interval is set to 10 [nsec] x 500 [cnt] + 100 [nsec] x 4000 [cnt] + 1000 [nsec] x 500 [cnt]), a shorter detection interval is set to 10 [nsec] x 4000 [cnt] + 100 [nsec] x 500 [cnt] + 1000 [nsec] x 500 [cnt]), and a combination of long and short detection intervals is set to 10 [nsec] x 500 [cnt] + 100 [nsec] x 500 [cnt] + 1000 [nsec] x 4000 [cnt]).

[0116] As described above, by changing the mixture ratio of the number of detections (counts) of each of the long and short detection intervals, it is possible to control the step at each luminance level. For example, control is performed so that the number of detections of the relatively short detection interval is increased in a case where it is necessary to obtain a step on the relatively high luminance side, and the number of detections of the relatively long detection interval is increased in a case where it is necessary to obtain a step on the relatively low luminance side. In this way, the step in the corresponding luminance region is improved, and thus it is possible to improve the SNR.

[0117] <11. Tenth Embodiment> The tenth embodiment of the present technology is an example in which the step and low power consumption are prioritized in a scheme in which photon detection is performed in a plurality of detection intervals. Furthermore, the tenth embodiment of the present technology is also an example in which the entire process of performing wave detection of a signal from the pixel array unit 120 and controlling at least one of the detection interval and the number of detections of photons based on the wave detection result on the application processor side, which will be described later.

[0118] [Configuration Example of Light Detection Device] Figure 17 is a block diagram illustrating a configuration example of the light detection device 100 in the tenth embodiment of the present technology.

[0119] In the light detection device 100 in the tenth embodiment of the present technology, in the configuration including the pulse generator 110, the pixel array unit 120, the interface circuit 130, the signal processing unit 140, and the control unit 150, the pulse generator 110, the pixel array unit 120, and the interface circuit 130 are arranged on the sensor chip 300. Furthermore, the signal processing unit 140 including the image processing unit 160 and the feature extraction unit 170 and the control unit 150 constitute the application processor 600.

[0120] In the application processor 600, the feature extraction unit 170 includes the wave detection circuit 180 and the target sensor response curve design unit 610.

[0121] Based on a signal of the pixel array unit 120 that has been linearized as a linear signal proportional to luminance by the linearization circuit 161 in the image processing unit 160, the wave detection circuit 180 generates a histogram that expresses the photon detection frequency as a frequency number, and creates a frequency distribution of the photon rate (luminance distribution) from the histogram.

[0122] Based on the frequency distribution of the photon rate created by the wave detection circuit 180, the target sensor response curve design unit 610 creates a target sensor response curve, and provides information of the target sensor response curve as a feature signal to the control unit 150.

[0123] Once the control unit 150 receives the information of the sensor response curve targeted from the target sensor response curve design unit 610, the control unit 150 estimates the photon detection interval T p and the count value (count number) N max of the given photon detection interval. In other words, the control unit 150 has a function of calculating the detection interval T p and the photon count value (photon detection number) N max . Note that the total maximum count value N is N =∑N max .

[0124] The control unit 150 controls at least one of the detection interval and the detection number of photons by giving the estimated photon detection interval T p and the count value N max of the given photon detection interval to the pulse generator 110.

[0125] As described above, the application processor 600 including the image processing unit 160, the feature extraction unit 170, and the control unit 150 detects a feature signal in the imaging result of the pixel array unit 120 and outputs a signal based on the feature signal to control at least one of the detection interval or the detection number of photons of the light detecting element 211.

[0126] Subsequently, the control flow in the low-power consumption priority mode in the light detecting apparatus 100 in the tenth embodiment of the present technology will be described.

[0127] [Step gradation priority mode (1)] Figure 18 is a view showing the control flow in the step gradation priority mode (1) in the light detecting apparatus 100 in the tenth embodiment of the present technology.

[0128] In Figure 18 , a indicates the entire image of a specific scene. The wave detecting circuit 180 creates a frequency distribution of the photon rate from a histogram representing the photon detection frequency as the frequency. Figure 18 b in Figure 18 shows the frequency distribution of the photon rate. The target sensor response curve design unit 610 creates a target sensor response curve based on the frequency distribution of the photon rate. Specifically, the target sensor response curve design unit 610 creates a sensor response curve in which the slope of the count value is larger in a case where the frequency distribution of the photon rate is higher, and the slope of the count value is smaller in a case where the frequency distribution is lower. Figure 18 c in Figure 18 shows the sensor response curve targeted. Then, as shown in d in Figure 18 , the control unit 150 estimates the photon detection interval T pand the count value N for a given photon detection interval max .

[0129] As mentioned above, in the tone priority mode (1), by Figure 17 The control unit 150 shown in FIG. 150 controls the image by referring to a histogram of the brightness distribution (frequency distribution of photon rates) of the entire image of a specific scene. The control is performed to increase the gradation within the brightness range where a higher frequency distribution exists. In other words, the gradation of the photon count value (number of detections) relative to the scene brightness distribution is maximized. By performing control while prioritizing the gradation of the photon count value, the detection interval of photons and the gradation of the photon count value relative to the scene brightness distribution can be appropriately controlled, thereby maximizing the information of the scene.

[0130] [Tonal Priority Mode (2)] In the gradation priority mode (1), control is performed to maximize the photon count value (number of detections) with respect to the gradation of the brightness distribution in the entire image of a specific scene. Conversely, in the gradation priority mode (2), control is performed to maximize the photon count value (number of detections) with respect to the gradation of the brightness distribution of the target subject.

[0131] Figure 19 10 is a diagram showing a control flow of the gradation priority mode ( 2 ) in the light detection device 100 in the tenth embodiment of the present technology.

[0132] exist Figure 19 In FIG. 1 , a shows the region of the target subject (dashed line frame). The wave detection circuit 180 creates a frequency distribution of the photon rate based on a histogram representing the photon detection frequency as a frequency. Specifically, a frequency distribution of the photon rate weighted in the target subject region is created. Figure 19 Graph b in FIG. 1 shows the frequency distribution of the photon rate. The target sensor response curve design unit 610 creates a target sensor response curve based on the frequency distribution of the photon rate. Specifically, the target sensor response curve design unit 610 creates a sensor response curve in which the slope of the count value is larger when the frequency distribution of the photon rate is higher, and the slope of the count value is smaller when the frequency distribution is lower. Figure 19 c in Figure 2 shows the target sensor response curve. Then, as Figure 19 As shown in d in FIG, the control unit 150 estimates the photon detection interval T that realizes the sensor response curve as the target. p and the count value N for a given photon detection interval max .

[0133] As mentioned above, in the gradation priority mode (2), Figure 17Under the control of the control unit 150 shown, the brightness distribution (frequency distribution of the photon rate) histogram in the region of the subject as a target is referred to, and control is performed to maximize the gradation of the photon count value (detection number) with respect to the brightness distribution. By controlling with priority to the gradation of the photon count value in this way, the detection interval of the photon and the photon count value can be appropriately controlled for the target subject brightness distribution, thereby maximizing the subject information.

[0134] [Low power consumption priority mode] If the count in a given photon detection interval is increased, the gradation of the corresponding brightness region is increased, and the signal-to-noise ratio (SNR) is improved, but the power consumption increases due to the increase in the count value of the counter 220. In other words, there is a trade-off relationship between the signal-to-noise ratio and the power consumption. Therefore, in the low power consumption priority mode, control is performed in a range not lower than the minimum required signal-to-noise ratio in the scene brightness range to minimize the count value of the counter 220. Figure 17 Under the control of the control unit 150 shown, control is performed in a range not lower than the minimum required signal-to-noise ratio in the scene brightness range to minimize the count value of the counter 220.

[0135] Figure 20 is a diagram illustrating the low power consumption priority mode of the light detection device 100 in the tenth embodiment of the present technology.

[0136] The SNR curve of a single detection interval is expressed by the following expression. [Math. 1] In the above expression, f p represents the incident photon rate, T p represents the photon detection interval, and N max represents the photon detection number (count value of the counter).

[0137] Figure 20 The SNR curves of a plurality of detection intervals T p are shown, and in Figure 20 , the large single-dot chain line represents the SNR curve at T p = 25 [nsec], the large broken line represents the SNR curve at T p = 100 [nsec], the double-dot chain line represents the SNR curve at T p = 400 [nsec], the small single-dot chain line represents the SNR curve at T p = 1600 [nsec], the small broken line represents the SNR curve at T p = 6400 [nsec], and the dot line represents the SNR curve at T p = 25600 [nsec]. The solid line represents the multiple exposure mode.

[0138] In the low power consumption priority mode of the light detection device 100 in the tenth embodiment of the present technology, when Figure 17 Under the control of the control unit 150 shown, a plurality of detection intervals T are drawn. p When the SNR curve of the light detection device 100 is obtained, control is performed to minimize the count value (number of photon detections) of the counter 220 within a range not lower than the minimum SNR required within the scene brightness range. Therefore, the power consumption of the light detection device 100 can be suppressed. Figure 20 In the example shown, it is assumed that the minimum SNR required over the scene brightness range is 25 [dB].

[0139] <12. Eleventh Embodiment> The eleventh embodiment of the present technology is an example of controlling the shutter time in a scheme in which photon detection is performed at multiple detection intervals. In the shutter time control mode, control is performed to shift only the dynamic range, wherein the sensor response curve (tone) created as a target and based on the frequency distribution of the photon rate is fixed.

[0140] [Configuration Example of Photodetection Device] Figure 21 11 is a block diagram showing a configuration example of the light detection device 100 in the eleventh embodiment of the present technology.

[0141] In the light detection device 100 in the eleventh embodiment of the present technology, the pulse generator 110, the pixel array unit 120, and the interface circuit 130 are arranged on the sensor chip 300, which is in a configuration including the pulse generator 110, the pixel array unit 120, the interface circuit 130, the signal processing unit 140, and the control unit 150. In addition, the signal processing unit 140 including the image processing unit 160 and the feature extraction unit 170, and the control unit 150 constitute the application processor 600.

[0142] In the shutter time control mode of the eleventh embodiment of the present technology, feature extraction unit 170 includes only wave detection circuit 180 in application processor 600 because the gradation (the shape of the sensor response curve) is fixed. In other words, the eleventh embodiment is configured so that target sensor response curve design unit 610, which is included in feature extraction unit 170 in the tenth embodiment, is no longer required. In this way, the configuration of application processor 600 in the eleventh embodiment can be simplified compared to the tenth embodiment, and the amount of simplification is equivalent to the elimination of target sensor response curve design unit 610.

[0143] Based on the signal from pixel array unit 120, which has been linearized into a linear signal proportional to brightness by linearization circuit 161 in image processing unit 160, wave detection circuit 180 creates a histogram representing the frequency of photon detection and, based on this histogram, creates a frequency distribution of photon rates (brightness distribution) as signal statistics. Wave detection circuit 180 then provides the signal statistics after wave detection as a characteristic signal to control unit 150.

[0144] Once the control unit 150 receives the frequency distribution of the photon rate as a characteristic signal from the wave detection circuit 180, the control unit 150 estimates the photon detection interval T that realizes the fixed sensor response curve. p And the count value (number of counts) N within a given photon detection interval max In other words, the control unit 150 has the function of calculating the detection interval T p and the photon count value (number of photon detections) N max function.

[0145] Then, the control unit 150 executes to detect all photons at intervals T p Multiply the control by n equally. Figure 22 The left side shows the detection interval T at all photons p The count value before and the SNR characteristic relative to the input photon rate are equally multiplied by n (for example, n=0.1), and Figure 22 The right side shows the detection interval T at all photons p The count value after being equally multiplied by n and the SNR characteristics relative to the input photon rate.

[0146] In the shutter time control mode in the eleventh embodiment of the present technology, a plurality of detection intervals T p The settings are 25 [nsec] (large single-dot chain line in the figure), 100 [nsec] (large dashed line), 400 [nsec] (double-dot chain line), 1600 [nsec] (small single-dot chain line), 6400 [nsec] (small dashed line) and 25600 [nsec] (dotted line), which are the same as the low power priority mode in the tenth embodiment of the present technology in this respect.

[0147] Under the control of the control unit 150, all photon detection intervals T p After being equally multiplied by n, the dynamic range of the input photon rate will shift by a factor of 1 / n. For example, if all photon detection intervals are T p Multiplying by 0.1 evenly, we get Figure 22 As shown, the dynamic range of the input photon rate will be from 10 4 to 10 8 [Hz] offset to 10 5 to 109 [Hz].

[0148] As described above, in the eleventh embodiment of the present technology, in the shutter time control mode, control is performed so as to shift the dynamic range without changing the shape of the sensor response curve created based on the photon rate frequency distribution created by the wave detection circuit 180. Therefore, it is possible to shift the dynamic range only without changing the sensor response curve shape and to maintain the gradation fixed.

[0149] <13. Twelfth Embodiment> The above-described tenth embodiment of the present technology is an example in which the entire processing of controlling at least one of the photon detection interval and the number of photon detections based on the wave detection result from the signal wave of the pixel array unit 120 is performed on the application processor 600 side. On the other hand, the twelfth embodiment of the present technology is a variation of the tenth embodiment of the present technology in which the signal statistics obtained by the wave detection of the application processor 600 are output to the sensor chip 300 side, and the processing at and after the design of the response curve is performed on the sensor chip 300 side. [Configuration Example of Light Detection Apparatus] Figure 23 is a block diagram illustrating a configuration example of the light detection apparatus 100 in the twelfth embodiment of the present technology.

[0150] In the light detection apparatus 100 in the twelfth embodiment of the present technology, in the configuration including the pulse generator 110, the pixel array unit 120, the interface circuit 130, the image processing unit 160, the wave detection circuit 180, the target sensor response curve design unit 610, and the control unit 150, the image processing unit 160 and the wave detection circuit 180 constitute the application processor 600. Further, the pulse generator 110, the pixel array unit 120, the interface circuit 130, the target sensor response curve design unit 610, and the control unit 150 are arranged on the sensor chip 300 side.

[0151] As described above, the light detection apparatus 100 in the twelfth embodiment of the present technology is configured so that the signal statistics after the wave detection of the wave detection circuit 180 in the application processor 600 are output to the sensor chip 300 side, and the processing at and after the design of the sensor response curve in the target sensor response curve design unit 610 is performed on the sensor chip 300 side. With this configuration, it is also possible to perform similar processing to the two-level priority modes (1) and (2) and the low-power-consumption-priority mode in the twelfth embodiment of the present technology.

[0152] Note that, although the twelfth embodiment of the present technology has been described as a variation of the tenth embodiment of the present technology, it is also possible to configure the twelfth embodiment as a variation of the eleventh embodiment of the present technology. In this variation, the deletion of Figure 23The configuration of the target sensor response curve design unit 610 in the first embodiment.

[0153] <14. Thirteenth Embodiment> In the first embodiment described above, the linearization circuit 161 linearizes the signal from the pixel array unit 120. Under this configuration, if the control unit 150 attempts to control the detection interval or the number of detections in accordance with the scene to be imaged, the response characteristics of the pixel array unit 120 also change. Therefore, in order to perform linearization in accordance with the scene, the inverse function of the function indicating the response characteristics is required. The light detection device 100 in the thirteenth embodiment differs from the first embodiment in that a lookup table is generated using the inverse function.

[0154] Figure 24 is a block diagram showing a configuration example of the light detection device 100 in the thirteenth embodiment of the present technology. The light detection device 100 in the thirteenth embodiment differs from the first embodiment in that a lookup table generation unit 163 and a memory 164 are further provided inside the image processing unit 160.

[0155] The lookup table generation unit 163 is configured to create a lookup table that associates the photon count value N p of the pixel array unit 120 with the photon rate f cnt in accordance with the detection interval T p and the number of detections set by the control unit 150. Here, as shown in Figure 14 , the response characteristics of the pixel array unit 120 to the photon incidence are nonlinear. The function indicating the response characteristics can be expressed, for example, by the following expression. [mathematical expression 2] In the above expression, N max is the total count number, and is a value obtained by subtracting “1” from the set value of the number of detections in the case of counting from “0”.

[0156] The inverse function in expression 2 is expressed by the following expression. [mathematical expression 3]

[0157] The lookup table generation unit 163 calculates the photon rate f p for each count value N cnt using expression 3, generates a lookup table that associates the values, and causes the memory 164 to hold the lookup table. The linearization circuit 161 acquires the photon rate f p corresponding to the count value N cnt with reference to the lookup table held in the memory 164, and outputs the photon rate f pThe data is provided to the feature extraction unit 170 and the image signal generation unit 162. Thus, linearization is achieved.

[0158] Figure 25 13 is a diagram showing an example of a lookup table in the thirteenth embodiment of the present technology. For example, when "5000" is set as the number of detection times, the count value N cnt The range of is "0" to "4999". The lookup table generation unit 163 converts "0" to "4999", the total count number N max (4999 in this case) and the set detection interval T p Substitute each of them into Expression 3 and calculate "f0" to "f 4999 The photon rate f for each value in p Then, the lookup table generation unit 163 causes the memory 164 to store a lookup table that associates the count value with the photon rate.

[0159] The figure assumes that only one detection interval T is set p However, the control unit 150 may set a plurality of detection intervals in order to expand the dynamic range as described above. Figure 26 and Figure 27 An example of a method of generating a lookup table when a plurality of detection intervals are set will be described.

[0160] As an example, Figure 26 As shown, for example, assuming that three different detection intervals T are set p1 、T p2 and T p3 In addition, it is assumed that "8", "6" and "5" are set to the detection interval T p1 、T p2 and T p3 The corresponding number of detections.

[0161] With the detection interval T p1 The corresponding count value N cnt (1) The range is "0" to "7" and is related to the detection interval T p2 The corresponding count value N cnt (2) The range is from "0" to "5". p3 The corresponding count value N cnt (3) The range of is "0" to "4". According to Expression 3, the photon rate when the count value is "0" is "0" regardless of the detection interval. Therefore, the lookup table generation unit 163 calculates the photon rate for each count value "1" and thereafter for each detection interval using the inverse function of Expression 3.

[0162] As an example, as shown in a of the figure, "3" to "105" are calculated as the photon rates f p1 corresponding to the detection interval T p (1) , and "1.1" to "23" are calculated as the photon rates f p2 corresponding to the detection interval T p (2) . In addition, "0.3" to "5" are calculated as the photon rates f p3 corresponding to the detection interval T p (3) .

[0163] In Figure 27 , a shows an example of the photon rate per detection interval. In a of the figure, the vertical axis represents the detection interval, and the horizontal axis represents the photon rate. In addition, the black dots represent the calculated values of the photon rate f p (1) , and the gray dots represent the calculated values of the photon rate f p (2) . The white dots represent the calculated values of the photon rate f p (3) . The numerical values in the circles represent the corresponding count values.

[0164] As an example, as shown in b of the figure, the look-up table generation unit 163 sorts the set of photon rates calculated for each detection interval according to a predetermined rule (for example, in ascending order). The set of photon rates f p cat , for example, is expressed with the following expression. [Math. 4] In the above expression, the set from 0.3 to 105 corresponds to the photon rate f p (1) , and the set from 1.1 to 23 corresponds to the photon rate f p (2) . The set from 0.3 to 5 corresponds to the photon rate f p (3) .

[0165] The set of photon rates f p , for example, can be expressed with the following expression. [Math. 5]

[0166] Then, the look-up table generation unit 163 assigns a new count value to each of the sorted photon rates.

[0167] Since the total number of calculated values of the photon rate in this example is 16, count values from "1" to "16" are assigned. For example, the minimum value of the photon rate in Expression 5 is "0.3", and therefore this value is assigned a count value of "1". In addition, the maximum value of the photon rate in Expression 5 is "105", and therefore this value is assigned a count value of "16". The look-up table generation unit 163 generates a look-up table that associates count values with photon rates. However, since they themselves contain errors, it is preferable to correct the values of the photon rates using the Newton method.

[0168] When correction is performed by the Newton method, for example, the calculated value of Expression 5 is used as the initial value f p_0 . In addition, based on Expression 2, the following function f(x) is defined. [Equation 6] In the above expression, I is an integer that represents the number of set detection intervals, and is "3" in the example of Expression 4.

[0169] By differentiating Expression 6, the following expression is obtained. [Equation 7]

[0170] Then, the look-up table generation unit 163 repeats the following expression until t becomes a predetermined value (for example, "2") for each count value, thereby obtaining a converged value. [Equation 8] In the above expression, the value of Expression 5 is used as the initial value f p_0 when t is "0".

[0171] The look-up table generation unit 163 uses the converged value obtained by Expression 8 as the photon rate after correction for each count value, and updates the look-up table using this value.

[0172] As described above, according to the thirteenth embodiment of the present technology, the look-up table generation unit 163 generates a look-up table using an inverse function, and therefore the linearization circuit 161 can refer to this table to appropriately perform linearization according to a scene.

[0173] [Variation of the Thirteenth Embodiment] Although in the above-described thirteenth embodiment, the look-up table generation unit 163 calculates and sorts the photon rates for each detection interval, and assigns count values when a plurality of detection intervals are set, it is desirable to further reduce the amount of calculation. The photodetection device of the variation of the thirteenth embodiment differs from the first embodiment in that a plurality of pre-created look-up tables are combined.

[0174] Figure 28 is a diagram for explaining a method of generating a lookup table in the modification of the thirteenth embodiment of the present technology. A plurality of lookup tables of different detection intervals, which are generated in advance, are stored in the storage 164 as a combination source lookup table. In the diagram, a shows an example of the combination source lookup table when the detection interval T a and the number of detections "10000" are set. In the table, the photon rates corresponding to the count values from "0" to "9999" are defined as f a0 to f a9999 .

[0175] In the diagram, b shows an example of the combination source lookup table when the detection interval T b and the number of detections "10000" are set. In the table, the photon rates corresponding to the count values from "0" to "9999" are defined as f b0 to f b9999 . Note that the number of combination source lookup tables in the storage 164 is not limited to two, and can be three or more.

[0176] Once the control unit 150 sets the detection interval and the number of detections, the lookup table generating unit 163 reads a plurality of combination source lookup tables having detection intervals close to the set detection interval from the storage 164. Then, the lookup table generating unit 163 generates a new lookup table as a combination target lookup table by weighted addition of the photon rates of these combination source lookup tables. For example, assume that the detection interval set at the time of imaging is T c , and the following relational expression is created between T c , T a , and T b . T c = T a × α + T b × β... Expression 9 α + β = 1... Expression 10 In the above expression, α and β are real numbers, and are used as weights.

[0177] As an example, as shown in c in the diagram, for example, the lookup table generating unit 163 reads f a0 for the photon rate corresponding to the count value "0" from the table in a in the diagram, and reads f b0 from the table of b in the diagram. Then, the lookup table generating unit 163 weighted adds the photon rates with weights α and β, respectively, and regards the added value as the photon rate corresponding to the count value "0" in the combination source lookup table. Similar weighted addition is performed for the count value "1" and subsequent count values. In this way, since the combination source lookup table generated in advance is used in the modification of the thirteenth embodiment, the amount of calculation can be reduced compared to the thirteenth embodiment.

[0178] The number of detections in FIG. c, which is the combined source lookup table, can be greater than the number of detections set by the control unit 150. In this case, the lookup table generation unit 163 performs thinning on the rows (count values and photon rates) in FIG. c according to the set value.

[0179] For example, in a case where the number of detections, which is the combined source lookup table in FIG. c, is set to "10000" and the number of detections is "5000", half of the rows are thinned. In the figure, d shows an example of the table after thinning. The rows shown by the hatching in FIG. d are rows that have been thinned. For example, the rows of odd count values in FIG. c have been thinned. Then, as shown in the example in FIG. d, new count values from "0" to "4999" are assigned to the remaining rows.

[0180] The combined target lookup table in FIG. c is saved in the memory 164 without thinning; whereas the combined target lookup table in FIG. d is saved in the memory 164 with thinning. The linearization circuit 161 performs linearization with reference to the combined target lookup table.

[0181] Note that, in a case where the above-described weighted addition is performed, an approximate solution of the photon rate for each count value can also be obtained based on a plurality of inverse functions by a B bulb algorithm such as a bisection method.

[0182] In this way, according to the modification of the thirteenth embodiment of the present technology, the lookup table generation unit 163 acquires the combined target lookup table by performing weighted addition on a plurality of combined source lookup tables, and thus it is possible to reduce the amount of calculation.

[0183] <15. Modification of each embodiment> Note that the above-described embodiments only show examples for implementing the present technology, and matters in the embodiments have a corresponding relationship with matters in the claims that define the invention. Also, matters in the claims that define the invention also have a corresponding relationship with matters in the embodiments of the present technology that have the same name. However, the present technology is not limited to these embodiments, and can be implemented by various modifications of these embodiments without departing from the gist thereof.

[0184] <16. Application example on a mobile body> The technology of the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure can be implemented as a device mounted on any type of mobile body (for example, a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, and a robot).

[0185] Figure 29 is a block diagram showing a schematic configuration example of a vehicle control system that is an example of a mobile body control system to which the technology according to the present disclosure can be applied.

[0186] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, a vehicle exterior information detection unit 12030, a vehicle interior information detection unit 12040, and an integrated control unit 12050. Further, a microcomputer 12051, a sound / image output section 12052, and a vehicle-mounted network interface (I / F) 12053 are shown as a functional configuration of the integrated control unit 12050. Figure 29 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, a vehicle exterior information detection unit 12030, a vehicle interior information detection unit 12040, and an integrated control unit 12050. Further, a microcomputer 12051, a sound / image output section 12052, and a vehicle-mounted network interface (I / F) 12053 are shown as a functional configuration of the integrated control unit 12050.

[0187] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle in accordance with various programs. For example, the drive system control unit 12010 functions as a control device for a drive force generation device (such as an internal combustion engine, a drive motor, or the like) for generating drive force of the vehicle, a drive force transmission mechanism for transmitting the drive force to wheels, a steering mechanism for adjusting the steering angle of the vehicle, a brake device for generating brake force of the vehicle, and the like.

[0188] The body system control unit 12020 controls the operation of various devices provided on the body of the vehicle in accordance with various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as a headlamp, a backup lamp, a brake lamp, a turn signal lamp, or a fog lamp. In this case, radio waves transmitted from a portable device that is a key substitute or signals of various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls a door lock device, a power window device, or a vehicle lamp of the vehicle, and the like.

[0189] The vehicle exterior information detection unit 12030 detects information about the outside of the vehicle including the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to an imaging section 12031. The vehicle exterior information detection unit 12030 causes the imaging section 12031 to image the outside of the vehicle, and receives the captured image. Based on the received image, the vehicle exterior information detection unit 12030 can perform processing for detecting a subject such as a person, a vehicle, an obstacle, a sign, or a character on a road surface, or processing for detecting the distance thereof.

[0190] The imaging section 12031 is an optical sensor that receives light and outputs an electric signal corresponding to the amount of received light. The imaging section 12031 can also output an electric signal as an image, or can output an electric signal as distance measurement information. In addition, the light received by the imaging section 12031 can be visible light, or can be invisible light such as infrared rays.

[0191] The in-vehicle information detecting unit 12040 detects information about the inside of the vehicle. For example, the in-vehicle information detecting unit 12040 is connected to a driver state detecting section 12041 that detects the state of the driver. The driver state detecting section 12041 includes, for example, a camera that images the driver. Based on the detection information input from the driver state detecting section 12041, the in-vehicle information detecting unit 12040 can calculate the degree of fatigue of the driver or the degree of concentration of the driver, or can determine whether the driver is dozing off.

[0192] The microcomputer 12051 can calculate a control target value of the driving force generating device, the steering mechanism, or the braking device based on information about the inside and outside of the vehicle obtained by the outside information detecting unit 12030 or the in-vehicle information detecting unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform coordinated control aimed at realizing functions of an advanced driver assistance system (ADAS), including collision avoidance or shock absorption of the vehicle, following driving based on the distance to the vehicle ahead, vehicle speed maintenance driving, vehicle collision warning, or vehicle lane departure warning, and the like.

[0193] In addition, based on the environmental information about the vehicle obtained by the outside information detecting unit 12030 or the in-vehicle information detecting unit 12040, the microcomputer 12051 can perform coordinated control for automatic driving that automatically drives the vehicle without relying on the operation of the driver or the like by controlling the driving force generating device, the steering mechanism, the braking device, or the like.

[0194] Further, the microcomputer 12051 can output a control command to the body system control unit 12030 based on the outside information about the vehicle obtained by the outside information detecting unit 12030. For example, the microcomputer 12051 can perform coordinated control aimed at preventing glare by controlling the vehicle headlamp to change from high beam to low beam, for example, in accordance with the position of the vehicle ahead or the oncoming vehicle detected by the outside information detecting unit 12030.

[0195] The sound / image output section 12052 transmits an output signal of at least one of sound and image to an output device capable of visually or aurally notifying information to the occupant of the vehicle or the outside of the vehicle. In Figure 29In the example of FIG. 12A, the audio speaker 12061, the display portion 12062, and the instrument panel 12063 are illustrated as output devices. The display portion 12062 may, for example, include at least one of a vehicle-mounted display and a head-up display.

[0196] Figure 30 FIG. 12B is a diagram illustrating an example of mounting positions of the imaging portions 12031.

[0197] In Figure 30 The imaging portions 12031 include imaging portions 12101, 12102, 12103, 12104, and 12105.

[0198] The imaging portions 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as a front nose, a rearview mirror, a rear bumper, a rear door, and an upper portion of a windshield of the vehicle 12100. The imaging portion 12101 provided at the front nose and the imaging portion 12105 provided at the upper portion of the windshield of the vehicle mainly acquire images of the front of the vehicle 12100. The imaging portions 12102 and 12103 provided at the rearview mirror mainly acquire images of both sides of the vehicle 12100. The imaging portion 12104 provided at the rear bumper or the rear door mainly acquires images of the rear of the vehicle 12100. The imaging portion 12105 provided at the upper portion of the windshield of the vehicle mainly functions to detect a preceding vehicle, a pedestrian, an obstacle, a signal lamp, a traffic sign, or a lane.

[0199] Note that Figure 30 Examples of imaging ranges of the imaging portions 12101 to 12104 are illustrated. An imaging range 12111 represents an imaging range of the imaging portion 12101 provided at the front nose. Imaging ranges 12112 and 12113 represent imaging ranges of the imaging portions 12102 and 12103 provided at the rearview mirror, respectively. An imaging range 12114 represents an imaging range of the imaging portion 12104 provided at the rear bumper or the rear door. A bird's-eye image of the vehicle 12100 is obtained, for example, by superimposing image data imaged by the imaging portions 12101 to 12104.

[0200] At least one of the imaging portions 12101 to 12104 can have a function for obtaining distance information. For example, at least one of the imaging portions 12101 to 12104 can be a stereo camera constituted by a plurality of imaging elements, or can be an imaging element having pixels for phase difference detection.

[0201] For example, the microcomputer 12051 can determine the distance of each three-dimensional subject within the imaging ranges 12111 to 12114 and the time change of the distance (relative speed with respect to the vehicle 12100) on the basis of the distance information obtained from the imaging sections 12101 to 12104, and extract, as a preceding vehicle, a closest three-dimensional subject that is present particularly on a travel path of the vehicle 12100 and travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, equal to or greater than 0 km / h). Furthermore, the microcomputer 12051 can be able to set in advance a following distance that should be kept in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. Thus, it is possible to perform coordinated control for automatic driving to automatically travel the vehicle without relying on the operation of the driver or the like.

[0202] For example, the microcomputer 12051 can classify three-dimensional subject data of three-dimensional subjects on the basis of distance information obtained from the imaging sections 12101 to 12104 into a two-wheeled vehicle, a standard-size vehicle, a large-size vehicle, a pedestrian, a utility pole, and other three-dimensional subjects, extract the classified three-dimensional subject data, and use the extracted three-dimensional subject data for automatic obstacle avoidance. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that can be visually identified by a driver of the vehicle 12100 and obstacles that are difficult to be visually identified by the driver of the vehicle 12100. Then, the microcomputer 12051 determines a collision risk indicating a risk of collision with each obstacle. In a case where the collision risk is equal to or higher than a set value and thus there is a possibility of collision, the microcomputer 12051 outputs a warning to the driver through the audio speaker 12061 or the display unit 12062, and performs forced deceleration or avoidance steering through the drive system control unit 12010. The microcomputer 12051 can thereby assist the driver to avoid collision.

[0203] At least one of the imaging sections 12101 to 12104 can be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the captured images of the imaging sections 12101 to 12104. Such pedestrian recognition is performed, for example, by a process of extracting feature points in the captured images of the imaging sections 12101 to 12104 as infrared cameras and a process of determining whether the pedestrian is a pedestrian by performing pattern matching processing on a series of feature points representing the outline of the subject. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging sections 12101 to 12104 and thus recognizes the pedestrian, the sound / image output section 12052 controls the display section 12062 so as to display a square outline for emphasis superimposed on the recognized pedestrian. In addition, the sound / image output section 12052 can control the display section 12062 so as to display an icon or the like for indicating the pedestrian at a desired position.

[0204] Thus far, an example of a vehicle control system to which the technology according to the present disclosure is applicable has been described. The technology according to the present disclosure is applicable to, for example, each imaging unit 12031 among the components described above. Specifically, the light detection device in each embodiment of the technology according to the present disclosure is applicable to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, good input / output characteristics of the light detection device can be achieved, and the performance of the moving body control system can be improved.

[0205] <16. Configuration applicable to related art> The present technology can also be configured as follows: (1) A light detection device comprising: a light detection element that detects the incidence of a photon; an output unit that outputs an imaging result of the light detection element; a charging unit that charges the light detection element; and a control unit that controls the charging unit based on the imaging result, with respect to at least one of a detection interval and a number of detections of the photon by the light detection element. (2) The light detection device according to the above (1), comprising: a pixel array unit in which the light detection element is arranged in an array; a setting unit that sets the detection interval and the number of detections of the photon; an image processing unit that generates an image signal based on a signal of the pixel array unit output from the output unit as the imaging result; and a feature extraction unit that extracts a feature signal based on the imaging result, The control unit controls setting information of the setting unit based on the feature signal extracted by the feature extraction unit. (3) The light detection device according to the above (2), wherein the feature signal is a dynamic range based on a histogram detection of a detection frequency of the photons, a maximum or minimum value of an abscissa of the histogram, a median of average luminance values, or a maximum detection frequency. (4) The light detection device according to the above (3), wherein the image processing unit includes a linearization circuit in an input stage, and the feature extraction unit extracts the feature signal based on a signal of the pixel array unit after passing through the linearization circuit. (5) The light detection device according to the above (4), wherein the pixel array unit outputs a count value of the photons, and the image processing unit includes: a lookup table generation unit that generates a lookup table associating the count value with a photon rate based on the detection interval and the number of detections, and the linearization circuit that converts the count value into the photon rate with reference to the lookup table and provides the photon rate to the feature extraction unit. (6) The light detection device according to the above (5), wherein the lookup table generation unit generates the lookup table using an inverse function of a function representing a response characteristic of the pixel array unit in response to an input of the photons. (7) The light detection device according to the above (6), wherein the setting unit sets two or more detection intervals, and the lookup table generation unit generates the lookup table by calculating the photon rate for each of the count values using the inverse function for each of the detection intervals, sorting a group of the photon rates calculated in accordance with a predetermined rule, and assigning the count value to each of the photon rates sorted. (8) The light detection device according to the above (6), wherein the setting unit sets two or more detection intervals, and the lookup table includes a plurality of combination source lookup tables and a combination target lookup table different in the detection interval, the lookup table generation unit generates the combination target lookup table by weightedly adding the photon rates in each of the plurality of combination source lookup tables, and the linearization circuit refers to the combination target lookup table. (9) The light detecting apparatus according to the above (4), wherein the pixel array unit, the setting unit, and the output unit are arranged on a first chip, and the image processing unit and the feature extraction unit are arranged on a second chip. (10) The light detecting apparatus according to the above (9), wherein the pixel array unit, the setting unit, and the output unit are arranged on the first chip, the image processing unit and the feature extraction unit are arranged on the second chip, and the control unit is arranged on a third chip. (11) The light detecting apparatus according to the above (9), wherein the pixel array unit, the setting unit, the output unit, and the control unit are arranged on the first chip, and the image processing unit and the feature extraction unit are arranged on the second chip. (12) The light detecting apparatus according to the above (3), wherein the feature extraction unit extracts the feature signal based on an image signal generated by the image processing unit. (13) The light detecting apparatus according to the above (12), wherein the pixel array unit, the setting unit, and the output unit are arranged on a first chip, the image processing unit is arranged on a second chip, and the control unit and the feature extraction unit are arranged on a third chip. (14) The light detecting apparatus according to the above (3), wherein the feature extraction unit includes a linearization circuit in an input stage, and extracts the feature signal based on a signal of the pixel array unit after passing through the linearization circuit. (15) The light detecting apparatus according to the above (14), wherein the pixel array unit, the setting unit, the output unit, the control unit, and the feature extraction unit are arranged on a first chip, and the image processing unit is arranged on a second chip. (16) The light detecting apparatus according to the above (3), wherein the control unit controls the setting information of the setting unit based on an external control of a user. (17) The light detecting apparatus according to the above (3), wherein the control unit sets a plurality of detection intervals as the detection interval of the photons, and performs the detection of the photons with the plurality of detection intervals. (18) The light detecting apparatus according to the above (17), wherein the control unit performs control based on a histogram of a luminance distribution in an entire image of a scene to maximize the number of detections of the photons with respect to a gradation of the luminance distribution. (19) The light detecting apparatus according to the above (17), wherein the control unit performs control based on a histogram of a luminance distribution in a region of a subject that is a target to maximize the number of detections of the photons with respect to a gradation of the luminance distribution. (20) The light detecting apparatus according to the above (17), wherein the control unit performs control to minimize the number of detections of the photons in a range that is not lower than a minimum SNR required in a scene luminance range when a signal-to-noise ratio (SNR) curve of the plurality of detection intervals is plotted. (21) The light detecting apparatus according to the above (17), wherein the control unit performs control to shift a dynamic range only without changing a shape of a sensor response curve created based on a photon rate frequency distribution created by the feature extraction unit. (22) The light detecting apparatus according to the above (17), wherein the feature extraction unit includes a wave detection circuit that creates a frequency distribution of a photon rate based on the imaging result, and a target sensor response curve design unit that creates a sensor response curve that is a target based on the frequency distribution of the photon rate, the image processing unit, the wave detection circuit, the target sensor response curve design unit, and the control unit constitute an application processor, and the application processor outputs a signal for controlling at least one of a detection interval and a number of detections of the photons to a sensor chip in which the pixel array unit is formed. (23) The light detecting apparatus according to the above (17), wherein the feature extraction unit includes a wave detection circuit that creates a frequency distribution of a photon rate based on the imaging result, and a target sensor response curve design unit that creates a sensor response curve that is a target based on the frequency distribution of the photon rate, the image processing unit, the wave detection circuit, and the control unit constitute an application processor, and the application processor outputs a signal for controlling at least one of a detection interval and a number of detections of the photons to a sensor chip in which the pixel array unit and the target sensor response curve design unit are formed. (24) An application processor that uses an imaging result output from a light detecting apparatus as input, the light detecting apparatus including: a light detecting element that detects incidence of a photon; an output unit that outputs an imaging result of the light detection element; a charging unit that charges the light detection element; and a control unit that controls the charging unit based on the imaging result, for at least one of a detection interval and a detection number of the photons detected by the light detection element, the application processor detects a characteristic signal in the imaging result, and outputs a signal for controlling at least one of a detection interval and a detection number of the photons detected by the light detection element based on the characteristic signal. List of Reference Signs

[0206] 100 light detection device 110 pulse generator 120 pixel array unit 130 interface circuit 140 signal processing unit 150 control unit 160 image processing unit 161, 190 linearization circuit 162 image signal generation unit 163 lookup table generation unit 164 memory 170 feature extraction unit 180 wave detection circuit 200 read circuit 210 pixel circuit 211 light detection element 212 clamping transistor 213 inverter 214 charging transistor 220 counter 230 selection switch 240 detection circuit 300 sensor chip 400 ISP chip 500 control chip 600 application processor 610 target sensor response curve design unit

Claims

1. A light detection device, comprising: a light detecting element that detects incident photons; an output unit configured to output an imaging result of the light detecting element; a charging unit configured to charge the light detecting element; as well as A control unit controls the charging unit with respect to at least one of a detection interval and a number of detection times of the photons detected by the light detecting element based on the imaging result.

2. The light detection device according to claim 1, comprising: a pixel array unit, wherein the light detection elements are arranged in an array; a setting unit, which sets the detection interval and the number of detections of the photons; an image processing unit that generates an image signal based on the signal of the pixel array unit output from the output unit as the imaging result; and a feature extraction unit, which extracts feature signals based on the imaging result, The control unit controls the setting information of the setting unit based on the feature signal extracted by the feature extraction unit.

3. The light detection device according to claim 2, wherein The characteristic signal is based on a dynamic range of histogram detection expressing the detection frequency of the photons as a frequency, a maximum value or a minimum value of the horizontal axis of the histogram, an average brightness value, or a median of the maximum detection frequency.

4. The light detection device according to claim 3, in, The image processing unit includes a linearizer in the input stage, and The feature extraction unit extracts the feature signal based on the signal of the pixel array unit after passing through the linearization circuit.

5. The light detection device according to claim 4, in, The pixel array unit outputs the count value of the photons, and The image processing unit includes: a lookup table generating unit that generates a lookup table that associates the count value with a photon rate based on the detection interval and the number of detections, and The linearization circuit converts the count value into the photon rate with reference to the lookup table and provides the photon rate to the feature extraction unit. The light detection device according to claim 5 , wherein: The lookup table generation unit generates the lookup table using an inverse function of a function representing a response characteristic of the pixel array unit in response to the input of the photon.

7. The light detection device according to claim 6, in, The setting unit sets two or more detection intervals, and The lookup table generating unit generates the lookup table by calculating the photon rate of each count value using the inverse function for each detection interval, sorting the calculated set of photon rates according to a predetermined rule, and assigning the count value to each sorted photon rate.

8. The light detection device according to claim 6, in, The setting unit sets two or more detection intervals, and The lookup table includes a plurality of combined source lookup tables and a combined target lookup table with different detection intervals. The lookup table generation unit generates the combined target lookup table by weighted addition of the photon rates in each of the plurality of combined source lookup tables, and The linearization circuit references the combined target lookup table.

9. The light detection device according to claim 4, in, The pixel array unit, the setting unit and the output unit are arranged on a first chip, and The control unit, the image processing unit, and the feature extraction unit are arranged on a second chip.

10. The light detection device according to claim 9, in, The pixel array unit, the setting unit and the output unit are arranged on the first chip, The image processing unit and the feature extraction unit are arranged on the second chip, and The control unit is arranged on a third chip.

11. The light detection device according to claim 9, in, The pixel array unit, the setting unit, the output unit and the control unit are arranged on the first chip, and The image processing unit and the feature extraction unit are arranged on the second chip.

12. The light detection device according to claim 3, wherein The feature extraction unit extracts the feature signal based on the image signal generated by the image processing unit.

13. The light detection device according to claim 12, in, The pixel array unit, the setting unit and the output unit are arranged on a first chip, The image processing unit is arranged on the second chip, and The control unit and the feature extraction unit are arranged on a third chip.

14. The light detection device according to claim 3, wherein The feature extraction unit includes a linearization circuit in an input stage, and extracts the feature signal based on a signal of the pixel array unit after passing through the linearization circuit.

15. The light detection device according to claim 14, in, The pixel array unit, the setting unit, the output unit, the control unit and the feature extraction unit are arranged on a first chip, and The image processing unit is arranged on the second chip.

16. The light detection device according to claim 3, wherein The control unit controls setting information of the setting unit based on external control by a user.

17. The light detection device according to claim 3, wherein The control unit sets a plurality of detection intervals as detection intervals of the photons, and performs detection of the photons at the plurality of detection intervals.

18. The light detection device according to claim 17, wherein The control unit performs control based on a histogram of brightness distribution in an entire image of a scene to maximize the number of detections of the photons with respect to a gradation of the brightness distribution.

19. The light detection device according to claim 17, wherein The control unit performs control based on a histogram of luminance distribution in a region of a target object so as to maximize the number of detections of the photons with respect to a gradation of the luminance distribution.

20. The light detection device according to claim 17, wherein The control unit performs control to minimize the number of detections of the photons within a range not lower than a minimum SNR required within a scene brightness range when plotting a signal-to-noise ratio (SNR) curve of the plurality of detection intervals.

21. The light detection device according to claim 17, wherein The control unit performs control to shift only the dynamic range without changing the shape of a sensor response curve created based on the photon rate frequency distribution created by the feature extraction unit.

22. The light detection device according to claim 17, in, The feature extraction unit includes: a wave detection circuit that creates a frequency distribution of photon rates based on the imaging result; and a target sensor response curve design unit that creates a target sensor response curve based on the frequency distribution of the photon rates. The image processing unit, the wave detection circuit, the target sensor response curve design unit and the control unit constitute an application processor, and The application processor outputs a signal for controlling at least one of a detection interval and a detection number of the photons to a sensor chip in which the pixel array unit is formed.

23. The light detection device according to claim 17, in, The feature extraction unit includes: a wave detection circuit that creates a frequency distribution of photon rates based on the imaging result; and a target sensor response curve design unit that creates a target sensor response curve based on the frequency distribution of the photon rates. The image processing unit, the wave detection circuit and the control unit constitute an application processor, and The application processor outputs a signal for controlling at least one of a detection interval and a detection number of the photons to a sensor chip in which the pixel array unit and the target sensor response curve design unit are formed.

24. An application processor using as input an imaging result output from a light detection device, the light detection device comprising: a light detecting element that detects incident photons; an output unit configured to output an imaging result of the light detecting element; a charging unit configured to charge the light detecting element; as well as a control unit that controls the charging unit with respect to at least one of a detection interval and a number of detection times at which the light detecting element detects the photons based on the imaging result, The application processor detects a characteristic signal in the imaging result and outputs a signal for controlling at least one of a detection interval and a detection number of the photons by the light detecting element based on the characteristic signal.

Images