Light detection device and light detection system

Abstract

[Problem] To improve temporal-spatial resolution and generate an image having high image quality. [Solution] This light detection device comprises a plurality of pixels arranged in a two-dimensional direction, and a signal processing unit for generating an image on the basis of pixel signals outputted from each of the plurality of pixels. Each of the plurality of pixels includes a photoelectric conversion element for detecting incident photons, and a pixel circuit for generating the pixel signal on the basis of whether the photoelectric conversion element has detected a photon in each of a plurality of detection intervals arranged in order in a time axis direction. Each of the plurality of detection intervals has an encoded time width.

Description

Optical detection device and optical detection system

[0001] The present disclosure relates to optical detection devices and optical detection systems.

[0002] Photodetectors using a single photon avalanche diode (SPAD) that amplifies electrons like an avalanche from a single incident photon are known. SPADs can detect even weak light and can detect light faster than conventional complementary metal-oxide-semiconductor (CMOS) devices. One application example of a photodetector using a SPAD is a technology that expands the dynamic range by adjusting the length of the photon detection cycle according to the illuminance (see Patent Document 1).

[0003] JP 2023-059522 A

[0004] However, in Patent Document 1, the time resolution is reduced because the photon detection period is lengthened. Also, for dynamic subjects, artifacts occur according to the detection period pattern, degrading the image data.

[0005] Therefore, the present disclosure provides a light detection device and a light detection system that can improve the spatiotemporal resolution and generate high-quality images.

[0006] In order to solve the above problems, according to the present disclosure, there is provided a photodetection device comprising: a plurality of pixels arranged in a two-dimensional direction; and a signal processing unit that generates an image based on pixel signals output from each of the plurality of pixels, wherein each of the plurality of pixels has a photoelectric conversion element that detects incident photons, and a pixel circuit that generates the pixel signal based on whether or not the photoelectric conversion element detects a photon in each of a plurality of detection intervals that are arranged in sequence in the time axis direction, and each of the plurality of detection intervals has an encoded time width.

[0007] The apparatus may further include an optimization unit that optimizes the time width of each of the plurality of detection intervals.

[0008] The optimization unit may optimize a time width of each of the plurality of detection intervals so as to minimize information loss of the pixel signal.

[0009] The optimization unit may randomly set the time width of each of the plurality of detection intervals.

[0010] The plurality of detection intervals may be arranged without gaps in the time axis direction.

[0011] At least some of the plurality of detection intervals, two detection intervals adjacent to each other in the time axis direction, may be spaced apart in the time axis direction.

[0012] The pixel circuit may generate the pixel signal including the number of detection intervals in which photons are incident, out of the plurality of detection intervals.

[0013] The signal processing unit may include a compressed image generation unit that generates a compressed image that is compressed in the spatial and temporal directions based on the plurality of pixel signals output from the plurality of pixels, and a restored image generation unit that generates a restored image based on the compressed image, with improved spatiotemporal resolution compared to the compressed image.

[0014] The image processing apparatus may include an information storage unit that stores first information for setting the time width of each of the plurality of detection intervals, and an information setting unit that sets second information required to generate the restored image, wherein the compressed image generation unit generates the compressed image for one frame based on a plurality of the pixel signals for a plurality of frames that are generated at the plurality of detection intervals that are set based on the first information, and the restored image generation unit may generate the restored image from the compressed images for a plurality of frames based on the second information.

[0015] The information storage unit may store predetermined first information, and the information setting unit may set predetermined second information.

[0016] The image processing device may further include a scene detection unit that detects characteristics of the compressed image, and an information determination unit that determines the first information and the second information based on the characteristics of the compressed image detected by the scene detection unit.

[0017] The information determination section may determine the first information and the second information based on characteristics including at least one of a type of subject included in the compressed image and illuminance.

[0018] The photoelectric conversion element may include a SPAD (Single Photon Avalanche Diode).

[0019] The photoelectric conversion element may be provided for each jot of a QIS (Quanta Image Sensor).

[0020] The present disclosure also provides a photodetection system including: a photodetection device that generates a plurality of pixel signals by photoelectric conversion; and a signal processing device that generates an image based on the plurality of pixel signals, wherein the photodetection device has a plurality of pixels that are arranged in a two-dimensional direction and generate the plurality of pixel signals, each of the plurality of pixels having a photoelectric conversion element that detects incident photons, and a pixel circuit that generates the pixel signal based on whether the photoelectric conversion element detects a photon in each of a plurality of detection intervals that are arranged in sequence in a time axis direction, and each of the plurality of detection intervals has an encoded time width.

[0021] The photodetector may have a compressed image generator that generates a compressed image that is compressed in the spatial and temporal directions based on the plurality of pixel signals, and the signal processing device may have a restored image generator that generates a restored image based on the compressed image, with improved spatiotemporal resolution compared to the compressed image.

[0022] The signal processing device may have an information storage unit that stores first information for setting the time width of each of the plurality of detection intervals; an information setting unit that sets second information required to generate the restored image; a scene detection unit that detects features of the compressed image; and an information determination unit that determines the first information and the second information based on the features of the compressed image detected by the scene detection unit.

[0023] The light detection device may include an information storage unit that stores first information for setting the time width of each of the plurality of detection intervals; an information setting unit that sets second information required to generate the restored image; a scene detection unit that detects features of the compressed image; and an information determination unit that determines the first information and the second information based on the features of the compressed image detected by the scene detection unit.

[0024] The signal processing device may include a signal processing unit that performs predetermined signal processing based on the restored image.

[0025] The photodetector and the signal processor may be configured on different semiconductor chips.

[0026] 10B is a block diagram showing a configuration of a photodetector according to a first embodiment of the present disclosure; FIG. 10C is a diagram showing a first example of a stacked structure of a photodetector; FIG. 10D is a diagram showing a second example of a stacked structure of a photodetector; FIG. 10E is a block diagram showing a configuration of a pixel according to a first embodiment of the present disclosure; FIG. 10F is a circuit diagram showing a configuration of a light receiving unit according to a first embodiment of the present disclosure; FIG. 10G is a cross-sectional view of a photoelectric conversion element according to a first embodiment of the present disclosure; FIG. 10H is a timing chart of each signal of a pixel according to a first embodiment of the present disclosure; FIG. 10I is a diagram showing photon detection during a measurement period of a pixel according to a first embodiment of the present disclosure; FIG. 10I is a diagram showing characteristics of a long detection interval and a short detection interval in an imaging scene with low illuminance; FIG. 10I is a diagram showing characteristics of a long detection interval and a short detection interval in an imaging scene with high illuminance; FIG. 10I is a diagram showing a measurement period in which a long detection interval and a short detection interval are combined; FIG. 10I is a graph showing the relationship between illuminance and the number of detected photons for each detection interval; FIG. 10I is a diagram showing a periodic measurement period; FIG. 10A is a diagram showing an artifact caused by the measurement period of FIG. 10A; and FIG. 10B is an enlarged view of the artifact of FIG. 10B. FIG. 10I is a flowchart of image generation of a photodetector according to a first embodiment of the present disclosure; FIG. 10I is a diagram showing a first configuration example of a measurement period according to a first embodiment of the present disclosure; and FIG. 10I is a diagram showing a second configuration example of a measurement period according to a first embodiment of the present disclosure. FIG. 1 is a diagram showing a spatiotemporal compressed image according to a first embodiment of the present disclosure. FIG. 2 is a diagram showing a restored image according to a first embodiment of the present disclosure. FIG. 3 is a diagram explaining machine learning of a plurality of detection intervals according to a first embodiment of the present disclosure. FIG. 4 is a diagram explaining machine learning of a restoration parameter according to a first embodiment of the present disclosure. FIG. 5 is a block diagram showing the configuration of a machine learning unit according to a first embodiment of the present disclosure. FIG. 6 is a block diagram showing a light detection system according to a first embodiment of the present disclosure. FIG. 7 is a diagram showing an imaging operation of a light detection device according to a first comparative example. FIG. 8 is a diagram showing a compressed image and a restored image according to a first comparative example. FIG. 9 is a diagram showing an imaging operation of a light detection device according to a second comparative example. FIG. 10 is a schematic diagram showing a measurement period of a light detection device according to a second comparative example. FIG. 11 is a block diagram showing a first configuration example of a light detection system according to a second embodiment of the present disclosure. FIG. 12 is a block diagram showing a second configuration example of a light detection system according to a second embodiment of the present disclosure. FIG. 13 is a diagram explaining machine learning of a detection interval according to a second embodiment of the present disclosure. FIG. 14 is a diagram explaining machine learning of a restoration parameter according to a second embodiment of the present disclosure.Fig. 10 is a block diagram showing a configuration of a pixel array unit according to a third embodiment of the present disclosure. Fig. 11 is a diagram explaining the operation of a jot according to a third embodiment of the present disclosure. Fig. 12 is a block diagram showing an example of a schematic configuration of a vehicle control system. Fig. 13 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit.

[0027] Hereinafter, embodiments of a light detection device and a light detection system will be described with reference to the drawings. The following description will focus on the main components of the light detection device and the light detection system, but the light detection device and the light detection system may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0028] First Embodiment Fig. 1 is a block diagram showing a configuration of a photodetector 10 according to a first embodiment of the present disclosure. The photodetector 10 detects incident light and outputs an image based on the incident light, and is mounted on, for example, a smartphone, a digital camera, a personal computer, an in-vehicle camera, an IoT (Internet of Things) camera, or the like. The photodetector 10 includes a pixel array unit 11, a signal processing unit 12, and a control unit 13. As will be described later, some of the components of the photodetector 10 shown in Fig. 1 may be provided in a device separate from the photodetector 10 (for example, a signal processing device, etc., which will be described later).

[0029] The pixel array unit 11 has a plurality of pixels 20 arranged in a first direction X and a second direction Y. In this specification, the left-right (horizontal) direction in Fig. 1 is referred to as the first direction X, and the up-down (vertical) direction in Fig. 1 is referred to as the second direction Y. Also, in this specification, a row of pixels 20 arranged in the first direction X is referred to as a pixel row. A column of pixels 20 arranged in the second direction Y is referred to as a pixel column.

[0030] The pixel 20 has a photoelectric conversion element that detects incident photons and generates a pixel signal based on the photons detected during a predetermined measurement period. The pixel signal includes, for example, grayscale information corresponding to the amount of incident light. Alternatively, the pixel signal may include event information indicating a change in the amount of incident light. A pixel 20 that outputs a pixel signal including grayscale information is called a grayscale pixel, and a pixel 20 that outputs a pixel signal including event information is called an EVS (Event-based Vision Sensor) pixel. The pixel array unit 11 may be composed of multiple grayscale pixels or multiple EVS pixels. Furthermore, the pixel array unit 11 may be configured to include both grayscale pixels and EVS pixels. Note that this specification describes an example in which the pixel array unit 11 is composed of multiple grayscale pixels.

[0031] A vertical signal line VSL is connected to each of the pixels 20 in each pixel column. Each vertical signal line VSL inputs pixel signals output from all the pixels 20 in the corresponding pixel column to the signal processing unit 12.

[0032] The signal processing unit 12 generates an image based on a plurality of pixel signals output from each of the plurality of pixels 20. The signal processing unit 12 includes a compressed image generating unit 14, a restored image generating unit 15, and an image processing unit 16.

[0033] The compressed image generation unit 14 generates a compressed image that is compressed in the spatial and temporal directions based on a plurality of pixel signals. The generated compressed image is an image that includes shaking (hereinafter referred to as blur) and has uneven blur. The compressed image generation unit 14 can also generate a compressed image for one frame based on pixel signals for a plurality of frames.

[0034] The restored image generation unit 15 generates a restored image having improved spatiotemporal resolution compared to the compressed image, based on the plurality of compressed images (blurred images) repeatedly generated by the compressed image generation unit 14. The restored image generation unit 15 can restore the original images of the plurality of frames from a compressed image of one frame generated based on pixel signals of the plurality of frames.

[0035] The processing of the image processing unit 16 can be performed by, for example, an ISP (Image Signal Processor). The image processing unit 16 performs predetermined image processing (e.g., demosaicing, distortion correction, etc.) on the restored image generated by the restored image generation unit 15. The image processing unit 16 outputs the processed image to a downstream application device, etc. As described above, the restored image generation unit 15 and the image processing unit 16 may be provided in a device (e.g., a signal processing device, which will be described later) separate from the photodetection device 10.

[0036] The control unit 13 selects pixel rows in order in synchronization with, for example, a vertical synchronization signal, and drives the pixels 20 for each pixel row. In this case, the pixels 20 output pixel signals in synchronization for each pixel row. Note that the pixels 20 may also be configured to output pixel signals asynchronously at individual timings. The control unit 13 is configured with a shift register, an arbiter, or the like.

[0037] The control unit 13 has an optimization unit 17. The optimization unit 17 adjusts the photon detection interval so as to minimize information loss of pixel signals when the compressed image is restored by the restored image generation unit 15 (or a downstream application processor, etc.). The detailed operation of the optimization unit 17 will be described later.

[0038] The photodetector 10 may include an information storage unit 18 that stores information (first information) for the optimization unit 17 to optimally drive the pixels 20, and an information setting unit 19 that sets information (second information) necessary to generate a restored image. The information storage unit 18 may be disposed within the control unit 13. The second information is also called a restoration parameter.

[0039] The photodetector 10 can be configured, for example, as a stacked chip in which multiple chips are stacked. FIG. 2A is a diagram showing a first example of the stacked structure of the photodetector 10. The photodetector 10 in FIG. 2A has a two-layer structure in which a pixel chip b1 and a logic chip b2 are bonded together in this order. These chips are bonded together by vias or the like. Note that the pixel chip b1 and the logic chip b2 may be bonded together by Cu-Cu bonding or bumps instead of vias.

[0040] The pixel chip b1 has, for example, a plurality of pixels 20 arranged in the pixel array unit 11. The logic chip b2 has, for example, a signal processing unit 12 and a control unit 13 arranged therein.

[0041] 2B is a diagram showing a second example of the stacked structure of the photodetector 10. The photodetector 10 in FIG. 2B has a three-layer structure formed by bonding a first pixel chip b3, a second pixel chip b4, and a logic chip b2 together in this order. Some of the components of the pixel 20 (e.g., photoelectric conversion elements and their peripheral circuits) are arranged on the first pixel chip b3. The remaining components of the pixel 20 are arranged on the second pixel chip b4. By arranging some of the components of the pixel 20 on the second pixel chip b2, the photodetector 10 can increase the proportion of the area of ​​the photoelectric conversion elements in the chip area, thereby improving sensitivity and enabling chip miniaturization.

[0042] The components arranged on each chip are not limited to those described above. The photodetector 10 may be configured with four or more stacked chips, or may be configured with a single flat chip.

[0043] 3 is a block diagram showing the configuration of the pixel 20 according to the first embodiment of the present disclosure. The pixel 20 includes a light receiving unit 21, a counter 22, and a selection switch 23.

[0044] The light receiving unit 21 generates a pulse signal PLS in response to incident photons and supplies the pulse signal PLS to the counter 22. The counter 22 counts the number of pulses in the pulse signal PLS. The count value of the counter 22 is initialized by a reset signal RSTc. The selection switch 23 switches whether or not to output a pixel signal Vimg based on the count value of the counter 22 based on the selection signal SEL. The selection signal SEL and the reset signal RSTc are supplied from, for example, the control unit 13.

[0045] FIG. 4 is a circuit diagram showing the configuration of the light receiving unit 21 according to the first embodiment of the present disclosure. The light receiving unit 21 includes a photoelectric conversion element 30, an inverter 31, a clipping transistor Tr1, a recharge transistor Tr2, and detection transistors Tr3 and Tr4. In the example shown in FIG. 4, the transistors Tr1, Tr2, and Tr3 are configured as NMOS (N-channel Metal-Oxide-Semiconductor) transistors. The transistor Tr4 is configured as a PMOS (P-channel Metal-Oxide-Semiconductor) transistor. The conductivity types of the transistors Tr1 to Tr4 can be changed as desired. Any one or all of the transistors Tr1 to Tr3 may be configured as NMOS transistors, and the transistor Tr4 may be configured as a PMOS transistor.

[0046] The photoelectric conversion element 30 is, for example, a SPAD. The photoelectric conversion element 30 detects photons incident on the pixel 20 and generates an electric charge. Either the anode or the cathode of the photoelectric conversion element 30 (the cathode in the example of FIG. 4) is connected to the drain of the transistor Tr1. The other of the anode or the cathode of the photoelectric conversion element 30 (the anode in the example of FIG. 4) is connected to a node having a voltage level lower than the power supply voltage node VDD.

[0047] The clip transistor Tr1 switches whether or not the photoelectric conversion element 30 and the detection node N1 are connected. The detection node N1 is a node connected to a detection circuit 32, which will be described later. The source of the clip transistor Tr1 is connected to the detection node N1. A clip signal CLIP is input to the gate of the clip transistor Tr1. The clip transistor Tr1 is turned on when, for example, a low-level clip signal CLIP is input.

[0048] The recharge transistor Tr2 switches whether or not to charge the photoelectric conversion element 30. The drain of the recharge transistor Tr2 is connected to the detection node N1. The source of the recharge transistor Tr2 is connected to the power supply voltage node VDD. A control signal XRST is input to the gate of the recharge transistor Tr2. The recharge transistor Tr2 is turned on when, for example, a low-level control signal XRST is input.

[0049] The detection transistors Tr3 and Tr4 constitute a detection circuit 32. The detection circuit 32 outputs a pulse signal PLS based on whether or not the photoelectric conversion element 30 detects a photon.

[0050] The drain of the detection transistor Tr3 is connected to the output node N2. The source of the detection transistor Tr3 is connected to the power supply voltage node VDD. The detection transistor Tr3 receives a detection signal Vdt from the detection node N1. The detection transistor Tr3 is turned on when the voltage level of the detection signal Vdt becomes equal to or lower than a predetermined threshold value.

[0051] The source of the detection transistor Tr4 is connected to the ground voltage node. The drain of the detection transistor Tr4 is connected to the output node N2. An inverted signal of the control signal XRST is input to the gate of the detection transistor Tr4 via an inverter 31. The detection transistor Tr4 is turned off when, for example, an inverted signal of the high-level control signal XRST is input.

[0052] The detection circuit 32 may have an inverter or the like that inverts the voltage of the output node N2, but this is not shown in FIG.

[0053] The clip signal CLIP and the control signal XRST are input from, for example, the control unit 13. Fig. 4 shows an example in which two transistors Tr2 and Tr4 are controlled by the control signal XRST. Note that the detection transistor Tr4 may be controlled independently by a signal different from the control signal XRST.

[0054] 5 is a cross-sectional view of a photoelectric conversion element 30 according to the first embodiment of the present disclosure. The photoelectric conversion element 30 has a light absorbing portion 41 and a charge multiplying portion 42. An on-chip lens 43 that focuses incident light is disposed on the light incident surface (upper side in FIG. 5 ) of the photoelectric conversion element 30. A color filter or the like that transmits only specific wavelength components of the incident light may be disposed between the photoelectric conversion element 30 and the on-chip lens 43, but is not shown in FIG. 5 .

[0055] The light absorbing unit 41 absorbs photons incident through the on-chip lens 43 and generates electric charges (e.g., electrons e). The charge multiplying unit 42 takes in the electrons e generated by the light absorbing unit 41, accelerates the electrons e using a high electric field, and causes the electrons e to collide with semiconductor atoms, generating new electrons e through ionization. In the charge multiplying unit 42, the generated electrons e collide with other semiconductor atoms, generating a large current through a chain reaction of impact ionization, a phenomenon known as avalanche multiplication. The photoelectric conversion element 30 can detect photons individually by multiplying each electron through avalanche multiplication.

[0056] 6A is a timing chart of each signal of the pixel 20 according to the first embodiment of the present disclosure, illustrating the clip signal CLIP, the control signal XRST, the cathode voltage Vpd of the photoelectric conversion element 30, the detection signal Vdt, and the pulse signal PLS.

[0057] First, at time t1, the clip signal CLIP goes high. This turns off the clip transistor Tr1, electrically disconnecting the photoelectric conversion element 30 from the detection node N1. This puts the photoelectric conversion element 30 into a standby state for the incidence of a photon.

[0058] At time t2, the control signal XRST goes high. This turns off the recharge transistor Tr2, disconnecting the detection node N1 from the power supply voltage node VDD. Also, the detection transistor Tr4 turns off, and the detection circuit 32 goes into a drive state.

[0059] At time t3, the photoelectric conversion element 30 detects a photon, and a large current is generated due to avalanche multiplication, causing the cathode voltage Vpd to drop significantly.

[0060] At time t4, the clip signal CLIP goes low. The clip transistor Tr1 is turned on, which transfers the charge in the photoelectric conversion element 30 to the detection node N1 and lowers the signal level of the detection signal Vdt. This turns on the detection transistor Tr3, causing the detection circuit 32 to output a high-level pulse signal PLS.

[0061] At time t5, the control signal XRST goes low, which turns on the recharge transistor Tr2, resets the voltage level of the detection node N1, and recharges the photoelectric conversion element 30. Also, the detection transistor Tr4 turns on, stops driving the detection circuit 32, and changes the pulse signal PLS to low.

[0062] At time t6, the recharging of the photoelectric conversion element 30 is completed, and the cathode voltage Vpd rises to the level before the photon was detected. Furthermore, the voltage level of the detection node N1 is reset, causing the signal level of the detection signal Vdt to rise and turning off the detection transistor Tr3. Accordingly, the clip signal CLIP goes high, and the photoelectric conversion element 30 waits for the incidence of a photon again.

[0063] As described above, the period (times t1 to t4) when the clip signal CLIP is at a high level is a waiting period for waiting for a photon to be incident on the photoelectric conversion element 30. The period when the clip signal CLIP is at a low level includes an output period (times t4 to t5) when a high-level pulse signal PLS is output, and a recharge period (times t5 to t6) until the photoelectric conversion element 30 is recharged.

[0064] During the standby period, the photoelectric conversion element 30 can detect one photon. After detecting a photon during the standby period, the photoelectric conversion element 30 cannot detect a new photon until it is recharged.

[0065] If the photoelectric conversion element 30 detects a photon during the standby period, a high-level pulse signal PLS is output during the output period. If the photoelectric conversion element 30 does not detect a photon during the standby period, a high-level pulse signal PLS is not output during the output period.

[0066] The photoelectric conversion element 30 is recharged during the recharge period, so that the photoelectric conversion element 30 can newly detect photons during the next standby period.

[0067] In this specification, the standby period, output period, and recharge period are collectively referred to as a detection interval. In the example of Fig. 6A, one photon is detected in each detection interval, but the pixel 20 may detect two or more photons in each detection interval.

[0068] FIG. 6B is a diagram showing photon detection during a measurement period of the pixel 20. In this specification, the period during which the number of photons for one frame is counted is referred to as the measurement period. As shown in FIG. 6B, the measurement period includes multiple detection intervals. In other words, the photoelectric conversion element 30 can detect multiple photons during the measurement period. FIG. 6B also shows the inversion voltage xVpd of the cathode voltage. The inversion voltage xVpd of the cathode voltage indicates whether or not a photon is detected for each detection interval.

[0069] 6B also illustrates a pulse signal PLS based on whether or not a photon is detected at each detection interval. The light receiving unit 21 in FIG. 3 outputs one pulse signal PLS when it detects a photon at each detection interval. The light receiving unit 21 does not output multiple pulse signals PLS during the same detection interval. Furthermore, if the light receiving unit 21 does not detect a photon during each detection interval, it does not output a pulse signal PLS.

[0070] 6B, the pulse signal PLS is output as a digital signal having one bit of information indicating whether or not a photon is detected at each detection interval. For example, the pulse signal PLS outputs 1 if a photon is detected and 0 if no photon is detected.

[0071] The counter 22 counts up each time the pulse signal PLS is output. By measuring the number of times the pulse signal PLS is output, the counter 22 can measure the number of detection intervals at which photons are detected within a measurement period. The pixel 20 can measure, for example, the luminance of a subject based on the number of photons measured within a predetermined measurement period.

[0072] As described above, the detection circuit 32 and the counter 22 generate the pixel signal Vimg based on the number of detection intervals in which photons are detected within the measurement period. That is, the pixel signal Vimg is generated based on whether or not the photoelectric conversion element 30 detected a photon in each of a plurality of detection intervals arranged in sequence along the time axis. In this specification, the detection circuit 32 and the counter 22 are collectively referred to as a pixel circuit. The pixel signal Vimg includes information regarding the number of detection intervals in which photons were detected within the measurement period. That is, the pixel circuit generates a pixel signal Vimg that includes the number of detection intervals in which photons were incident.

[0073] In the measurement period of Fig. 6B, detection intervals, which are exposure periods, are arranged without gaps along the time axis. Note that a non-exposure period may be provided between two detection intervals. The non-exposure period can be provided by, for example, shutter control.

[0074] The pixel 20 can change the length of the detection interval by changing the length of the period during which the CLIP signal is at high level. Hereinafter, a relatively long detection interval will be referred to as a long detection interval, and a relatively short detection interval will be referred to as a short detection interval.

[0075] Because the number of photons incident on the photoelectric conversion element 30 is enormous, the number of incident photons is estimated by counting a portion of the incident photons. This estimation method is also called the CLK-DET (Clocked Detector) method. The photodetector 10 uses the CLK-DET method to prevent overflow of the measurement value of the counter 22, etc.

[0076] In the CLK-DET method, photon count loss is intentionally generated. One method for generating photon count loss is to use a long detection interval, which lengthens the photon detection period. With a long detection interval, count loss can be generated by not detecting photons other than the first photon detected. Another method for generating photon count loss is to use a short detection interval, which shortens the photon detection period, to thin out photons.

[0077] 7A and 7B are diagrams illustrating the characteristics of the long and short detection intervals in a low-illuminance imaging scene. Fig. 7B is a diagram illustrating the characteristics of the long and short detection intervals in a high-illuminance imaging scene. Note that Figs. 7A and 7B show an example in which a non-exposure period is provided between two short detection intervals to thin out photons.

[0078] 7A, in a low-illuminance imaging scene where the number of incident photons is small, many of the photons are incident during the non-exposure period, and the incident photons often cannot be detected with a short detection interval. Therefore, in a low-illuminance imaging scene, it is desirable to use a long detection interval, which reduces the chance of missing photons.

[0079] 7B , in a high-illuminance imaging scene with many incident photons, photons are detected at almost all detection intervals with a long detection interval, and there is a risk that periods when no photons are incident will not be accurately detected. Therefore, in a high-illuminance imaging scene, it is desirable to use a short detection interval that can detect photons by thinning them out.

[0080] The photodetector 10 may also combine a long detection interval and a short detection interval. FIG. 8 is a diagram showing a measurement period in which a long detection interval and a short detection interval are combined. In the measurement period of FIG. 8 , in an image scene with high illuminance, photons can be thinned out and detected using the short detection interval. In addition, in the measurement period of FIG. 8 , in an image scene with low illuminance, photons can be prevented from being missed from being detected using the long detection interval.

[0081] 8 shows an example in which long detection intervals and short detection intervals are alternately arranged. However, the measurement period of the CLK-DET method may include a period in which two or more long detection intervals (or short detection intervals) continue, or the long detection intervals and short detection intervals may be arranged randomly, or the measurement period may be composed of three or more detection intervals. As will be described later, the photodetector 10 according to the first embodiment of the present disclosure further optimizes the arrangement of multiple detection intervals with different time widths in the measurement period of the CLK-DET method.

[0082] FIG. 9 is a graph showing the relationship between illuminance and the number of detected photons for each detection interval. The vertical axis of FIG. 9 represents the number of detected photons (counts), and the horizontal axis represents illuminance. FIG. 9 also shows a curve W1 representing the measurement results at a long detection interval, a curve W2 representing the measurement results at a short detection interval, and a curve W3 representing the measurement results combining the long and short detection intervals. As shown by curve W1, measurements using the long detection interval cannot ensure appropriate gradation in high-illuminance regions. As shown by curve W2, measurements using the short detection interval cannot ensure appropriate gradation in low-illuminance regions. As shown by curve W3, measurements combining the long and short detection intervals can ensure appropriate gradation over a wide illuminance range.

[0083] As shown by the curve W3 in FIG. 9 , the photodetector 10 according to the first embodiment of the present disclosure can provide appropriate gradations over a wide illuminance range and expand the dynamic range by combining a long detection interval and a short detection interval.

[0084] As shown in Figure 6B, only one photon can be detected in one detection interval, regardless of the length of the detection interval. Therefore, when a long detection interval is used, the temporal resolution of the generated image decreases. The compressed image generator 14 in Figure 1 generates an image based on the measurement value of the counter 22. When an image is generated based on the number of photons detected by any combination of multiple detection intervals with different time widths, image data (hereinafter referred to as a compressed image) compressed in the spatiotemporal direction can be generated, taking into account multiple pixel regions with different illuminances.

[0085] The compressed image described above can be restored to its original state using existing restoration algorithms. However, the restored original image suffers from image degradation due to the combination of multiple detection intervals with different time durations for photon detection.

[0086] 10A is a diagram showing an example of photon detection in which short detection intervals D1 and long detection intervals D2 are alternately arranged within a measurement period. The multiple short detection intervals D1 included in the measurement period have the same length, and the multiple long detection intervals D2 also have the same length.

[0087] Fig. 10B is a diagram showing a captured image of a disk-shaped rotating body captured using the measurement period of Fig. 10A. Fig. 10C is an enlarged view of range Zm of the captured image of Fig. 10B. As shown in Fig. 10C, when short detection intervals D1 and long detection intervals D2 are alternately repeated, streaks are periodically visible in the captured image. These streaks are called motion artifacts and are a cause of degradation in the image quality of the captured image.

[0088] As described above, imaging using the CLK-DET method generates characteristic motion artifacts (hereinafter, "blur") that depend on the length and arrangement of multiple detection intervals. The photodetector device 10 according to the first embodiment of the present disclosure is characterized by controlling the occurrence of blur in a compressed image by performing coded exposure using a combination of multiple detection intervals with different time widths, as described below. This makes it possible to generate a high-quality restored image that does not generate motion artifacts, as shown in FIG. 10C , when the compressed image is restored.

[0089] FIG. 11 is a flowchart of image generation by the photodetector 10 according to the first embodiment of the present disclosure. First, the optimizer 17 sets a detection interval for generating a compressed image (step S1). The detection interval is set based on, for example, first information stored in the information storage unit 18. The first information includes setting information for the time widths of each of a plurality of detection intervals. The information storage unit 18 may store predetermined first information. More specifically, the first information may be stored in the information storage unit 18 during the manufacturing stage of the photodetector 10, or may be stored in the information storage unit 18 before the processing of step S1 is performed. The control unit 13 may control the storage of the first information in the information storage unit 18, or an external device different from the photodetector 10 may control the storage of the first information in the information storage unit 18.

[0090] FIG. 12A is a diagram illustrating a first configuration example of a measurement period according to the first embodiment of the present disclosure. In step S1 of FIG. 11, multiple types of detection intervals (detection intervals D1, D2, D3, D4, D5, and D6 in the example of FIG. 12A) are determined. The measurement period illustrated in FIG. 12A includes multiple detection intervals D1 to D6. Each of the detection intervals D1 to D6 has a time width optimized by the optimization unit 17 (i.e., an encoded time width). In this specification, performing exposure at each of the multiple detection intervals encoded by the optimization unit 17 is referred to as coded exposure. Each detection interval is an exposure period, and imaging is performed by combining multiple exposure periods with different time widths.

[0091] The optimization unit 17 optimizes the time width of each of the plurality of detection intervals so as to minimize information loss in the pixel signal Vimg. For example, the optimization unit 17 adjusts the time width of the plurality of detection intervals so as to intentionally create unevenness in the blur of the compressed image. For example, by generating long detection intervals in areas where information in the pixel signal Vimg is sparse, information loss due to blur can be reduced. As will be described later, creating unevenness in the blur of the compressed image can improve the spatiotemporal resolution when generating a restored image.

[0092] The optimization unit 17 also optimizes the placement of the detection intervals D1 to D6. The detection intervals D1 to D6 are, for example, placed without periodicity (for example, randomly). The time widths and placements of the detection intervals D1 to D6 may be determined empirically or by machine learning.

[0093] In the measurement period of FIG. 12A , multiple detection intervals D1 to D6 are arranged without gaps in the time axis direction. A non-exposure period may be arranged between two detection intervals. FIG. 12B is a diagram showing a second configuration example of a measurement period according to the first embodiment of the present disclosure. The measurement period of FIG. 12B differs from the measurement period of FIG. 12A in that it includes one or more non-exposure periods. That is, in the measurement period of FIG. 12B , at least some of two detection intervals adjacent in the time axis direction are arranged spaced apart in the time axis direction. The duration and arrangement of the non-exposure periods may be random, or may be determined empirically or by machine learning.

[0094] The information on the plurality of detection intervals included in the first information is set within the scope of hardware constraints or design rule constraints, which may include, for example, a control method for the pixel array unit 11 (uniform control for all pixels, control by pixel row, control by pixel column, or control by pixel), the number of types of detection intervals that can be set, or the number of each detection interval.

[0095] Next, the information setting unit 19 sets restoration parameters in the restored image generating unit 15 (step S2). The information setting unit 19 may receive predetermined restoration parameters. Specifically, the information setting unit 19 may receive restoration parameters input during the manufacture of the photodetector 10, restoration parameters generated by machine learning, or restoration parameters input from an external device of the photodetector 10. The processing order of steps S1 and S2 is arbitrary, and steps S1 and S2 may be performed simultaneously.

[0096] Next, an image of the subject is captured by the plurality of pixels 20 in the pixel array unit 11 (step S3). The control unit 13 supplies a clip signal CLIP, a control signal XRST, and the like to each pixel 20 based on detection intervals D1 to D6 shown in Fig. 12A and the like, thereby controlling the operation of the pixels 20. Note that the time widths of the plurality of detection intervals D1 to D6 may be common to all pixels 20 in the pixel array unit 11, or detection intervals D1 to D6 with different time widths may be provided for each pixel 20.

[0097] Fig. 13A is a diagram showing the subject G1 photographed in step S3. The subject G1 is a moving object moving, for example, to the left in Fig. 13A.

[0098] The compressed image generation unit 14 generates a compressed image based on the multiple pixel signals Vimg output by the multiple pixels 20 (and pixel circuits) (step S4). FIG. 13B is a diagram showing an example of the compressed image G2 generated in step S4. The compressed image G2 is an image compressed in both the spatial and temporal directions. The compressed image G2 is generated by repeatedly exposing at multiple detection intervals with different time widths, and therefore blurring occurs. The blurring in the compressed image G2 occurs depending on the encoded time widths of the multiple detection intervals D1 to D6. In other words, the compressed image generation unit 14 generates a compressed image based on the multiple pixel signals Vimg generated at the multiple detection intervals set based on the first information.

[0099] In step S4 of Fig. 11, a compressed image for one frame may be generated based on pixel signals Vimg for multiple frames. This makes it possible to reduce the data rate between the compressed image generator 14 and the restored image generator 15 in Fig. 1. In this case, the restored image generator 15 generates restored images for multiple frames from the compressed image for one frame.

[0100] Next, the restored image generation unit 15 restores the compressed image G2 based on the restoration parameters set in step S2 to generate a restored image (step S5). Fig. 13C is a diagram showing the restored image G3 restored in step S5. The restored image generation unit 15 can restore the compressed image with uneven blur to restore the restored image G3 with improved spatiotemporal resolution.

[0101] The image processing unit 16 performs predetermined image processing on the restored image G3 to generate a restored image (step S6). Then, it is determined whether or not to end the shooting (step S7). If the shooting is to continue, the processing from step S3 onwards is repeated.

[0102] The optimization unit 17 can optimize the detection interval using, for example, a machine learning technique. FIG. 14A is a conceptual diagram illustrating an example in which the optimization unit 17 optimizes the detection interval using a machine learning technique. In the example of FIG. 14A , the optimization unit 17 has a learning model M1. During the learning stage of the learning model M1, image data G4 with multiple known detection intervals is used as input parameters for the learning model M1 to train the learning model M1. The image data G4 input to the learning model M1 during the learning stage of the learning model M1 is, for example, image data of a normal method captured by performing photoelectric conversion according to the amount of incident light in the same manner as a normal image sensor, without employing a CLK-DET method. The learning model M1 is composed of multiple layers, each of which has multiple nodes, and the nodes of two adjacent layers are connected by links. In the learning stage of the learning model M1, a combination of multiple image data G4 captured using a normal method for various subjects and multiple known detection intervals for each image data is used as training data, and the weights of each link in the learning model M1 are updated so that when each image data included in the training data is input into the learning model M1, multiple corresponding detection intervals are output.

[0103] By inputting image data captured using a normal method with unknown detection intervals into the learning model M1 that has completed the learning stage, it is possible to output a plurality of optimal detection intervals from the learning model M1.

[0104] Similarly, the information setting unit 19 can optimize the restoration parameters using, for example, a machine learning technique. FIG. 14B is a diagram illustrating machine learning of restoration parameters according to the first embodiment of the present disclosure. The learning model M2 shown in FIG. 14B is intended to infer restoration parameters using the detection interval and the compressed image G5 as input parameters. Note that either the detection interval or the compressed image G5 may be omitted from the input parameters. The learning model M2 comprises multiple layers, each of which has multiple nodes, and the nodes of two adjacent layers are connected by links. During the learning phase of the learning model M2, combinations of the detection interval, the compressed image G5, and known restoration parameters for restoring the compressed image G5 to a restored image G6 are input to the learning model M2 as training data, and the weights of each link of the learning model M2 are updated so that the known restoration parameters are output from the learning model M2. During the learning phase, the weights of the learning model M2 are repeatedly updated using multiple compressed images G5. In the learning stage, it is desirable to input various compressed images G5 with different types, sizes, brightness, and colors of the subject and background, and to train the learning model M2.

[0105] By inputting the detection interval and a compressed image with unknown restoration parameters to the learning model M2 that has completed the learning stage, the restoration parameters can be inferred.

[0106] FIG. 14C is a block diagram showing the configuration of a machine learning unit 50 that infers restoration parameters using the machine learning technique shown in FIG. 14B . The machine learning unit 50 is disposed in, for example, the information setting unit 19. As shown in FIG. 14C , the machine learning unit 50 includes a model construction unit 51, an information acquisition unit 52, and an inference unit 53. The model construction unit 51 constructs a learning model M2 through machine learning. The information acquisition unit 52 acquires multiple types of compressed images and multiple detection intervals corresponding to each compressed image. The inference unit 53 inputs the multiple detection intervals and compressed images acquired by the information acquisition unit 52 into the learning model M2, and infers restoration parameters for restoring a compressed image for which restoration parameters are unknown.

[0107] 14A , the machine learning unit 50 may infer the optimal detection intervals. In this case, the machine learning unit 50 is disposed in, for example, the optimization unit 17. The model construction unit 51 constructs a learning model M1, the information acquisition unit 52 acquires image data for which the multiple detection intervals are known, and the inference unit 53 infers the optimal detection intervals from the learning model M1.

[0108] FIG. 15 is a block diagram showing a photodetection system 1 according to a first embodiment of the present disclosure. The photodetection system 1 includes a photodetection device (sensor) 10a and a signal processing device (application processor) 2. The photodetection device 10a of FIG. 15 has a partially different internal configuration from the photodetection device 10 of FIG. 1. The photodetection device 10a of FIG. 15 omits a portion of the internal configuration of the signal processing unit 12 included in the photodetection device 10 of FIG. 1. Specifically, the restored image generation unit 15 and the image processing unit 16 included in the signal processing unit 12 of FIG. 1 are not provided in the photodetection device 10a of FIG. 15 but are provided in the signal processing device 2. Note that the photodetection device 10a and the signal processing device 2 may be provided on separate chips, for example, or may be integrated into a single chip.

[0109] The photodetector 10a outputs a compressed image G5 based on incident light. The photodetector 10a includes a pixel array unit 11, a compressed image generator 14, and a controller 13. The controller 13 includes a register 18a. The register 18a stores first information related to the detection interval, similar to the information storage unit 18 in FIG. 1 .

[0110] The signal processing device 2 generates a restored image G6 based on the compressed image G5 and performs predetermined image processing. The signal processing device 2 has a restored image generation unit 15, an image processing unit 16, and an information setting unit 19. The image processing unit 16 performs predetermined signal processing based on the restored image. The restored image generation unit 15 and the image processing unit 16 constitute a signal processing unit 12a. In addition to the above components, the signal processing device 2 may also have an image display unit, an output interface, etc. This allows the light detection system 1 to capture an image of the subject G7.

[0111] The photodetector 10a and the signal processing device 2 may be configured, for example, as different semiconductor chips. As a more specific example of implementation, the photodetector 10a and the signal processing device 2 may be arranged together in a single device (for example, a smartphone). In this case, the photodetector 10a is, for example, an image sensor, and the signal processing device 2 is, for example, an application processor.

[0112] Fig. 16A is a diagram showing the imaging operation of a photodetector according to a first comparative example. The photodetector according to the first comparative example differs from the photodetector 10 of Fig. 1 in that it does not perform coded exposure. In the measurement period of Fig. 16A, multiple exposure periods having a fixed time width are arranged without gaps. The measurement period of Fig. 16A is also called a long exposure period.

[0113] FIG. 16B shows a compressed image G101 and a restored image G102 according to the first comparative example. The compressed image G101 in FIG. 16B is blurred. The blur in the compressed image G101 is uniform because it is not controlled by the coded exposure. The restored image G102 restored from the compressed image G101 has deteriorated image quality due to information loss caused by the blur. For this reason, the photodetector according to the first comparative example cannot generate high-quality images.

[0114] FIG. 17A is a diagram showing the imaging operation of a photodetector according to a second comparative example. The photodetector according to the second comparative example uses shutter control to provide multiple non-exposure periods in a measurement period and encode the exposure period. Encoding the exposure period allows exposure to occur over exposure periods of various durations, resulting in uneven blurring of the captured image. FIG. 17B is a schematic diagram of the measurement period of the photodetector according to the second comparative example. As shown in FIG. 17B, a non-exposure period is always placed between two adjacent exposure periods during the measurement period according to the second comparative example, which reduces the amount of exposure light that can be captured during the measurement period and reduces the signal-to-noise (SN) ratio of the restored image.

[0115] The photodetector 10 according to the first embodiment of the present disclosure is capable of controlling blur in a compressed image by coded exposure, thereby minimizing information loss due to compression and decompression and outputting a high-quality image, as compared to the photodetector according to the first comparative example. Furthermore, the photodetector 10 is capable of increasing the exposure amount during a measurement period, as opposed to the photodetector according to the second comparative example, without necessarily providing a non-exposure period between two exposure periods (i.e., two detection intervals). This allows the photodetector 10 to output a decompressed image with a higher S / N ratio than the second comparative example.

[0116] As described above, in the first embodiment of the present disclosure, the detection intervals for the CLK-DET method are optimized, thereby generating a compressed image with uneven blur and improving the spatiotemporal resolution when restored. Furthermore, by arranging all detection intervals without gaps and eliminating non-exposure periods, the amount of exposure can be increased, thereby improving the S / N ratio. Furthermore, a compressed image for one frame can be generated based on pixel signals for multiple frames, and multiple restored images for multiple frames can be generated from the compressed image for one frame. This reduces the data rate between the compressed image generation unit 14 and the restored image generation unit 15, reducing the impact of noise and reducing hardware costs. This facilitates integration into mobile devices such as smartphones. As described above, the photodetector 10 according to the first embodiment of the present disclosure can improve spatiotemporal resolution and generate high-quality images.

[0117] Second Embodiment In a first embodiment of the present disclosure, a detection interval and restoration parameters are set in advance. The detection interval and restoration parameters may be dynamically determined depending on the imaging scene. FIG. 18A is a block diagram showing a first configuration example of a light detection system 1 according to a second embodiment of the present disclosure. The light detection system 1a shown in FIG. 18A includes a light detection device 10b. The light detection device 10b includes a scene detection unit 61 and an information determination unit 62.

[0118] The scene detection unit 61 acquires the compressed image G5 from the compressed image generation unit 14. The scene detection unit 61 detects features of the compressed image G5 (e.g., the type of subject or illuminance). The information determination unit 62 determines first information and second information (i.e., the detection interval and restoration parameters) based on the features of the compressed image G5 detected by the scene detection unit 61. In this way, the information determination unit 62 includes the function of the optimization unit 17 in FIG. 1 in order to perform processing to determine the detection interval. The information determination unit 62 inputs the first information to the register 18a and the second information to the information setting unit 19. The features of the compressed image G5 detected by the scene detection unit 61 are also called features of the captured scene.

[0119] Fig. 18B is a block diagram showing a second configuration example of the light detection system 1 according to the second embodiment of the present disclosure. The light detection system 1b shown in Fig. 18B differs from the light detection system 1a in Fig. 18A in that a scene detection unit 61 and an information determination unit 62 are arranged in a signal processing device 2a. Note that the scene detection unit 61 and the information determination unit 62 may be applied to the light detection device 10 in Fig. 1.

[0120] The information determination unit 62 may determine the detection interval or restoration parameters through machine learning. FIG. 19A is a diagram illustrating machine learning of the detection interval according to the second embodiment of the present disclosure. The learning model (second learning model) M3 shown in FIG. 19A is intended to infer an optimal detection interval using the features of the compressed image G5 detected by the scene detection unit 61 as input parameters. Like the learning model M2 described above, the learning model M3 is capable of updating the weights of links connecting nodes in two adjacent layers. During the learning phase of the learning model M3, features of compressed images with known detection intervals are input to the learning model M3, and the weights of each link are updated so that the corresponding detection interval is output. Multiple compressed images with known detection intervals corresponding to the features of various object images are input to the learning model M3, and the weights of the learning model M3 are repeatedly updated.

[0121] By inputting the compressed image features detected by the scene detection unit 61 for a new subject image into the learning model M3 that has completed the learning stage, the learning model M3 can output an optimal detection interval.

[0122] As shown in Figure 14A, the input parameters of the learning model M may be image data captured using a normal method, without using the CLK-DET method, by performing photoelectric conversion according to the amount of incident light, as with a normal image sensor.

[0123] The features of the subject may be the compressed image itself. That is, the learning model M3 may use the compressed image as an input parameter to infer a detection interval corresponding to the compressed image. In this case, if the data volume of the compressed image is large, the scene detection unit 61 may further compress the data before inputting it to the learning model M3.

[0124] FIG. 19B is a diagram illustrating machine learning of restoration parameters according to the second embodiment of the present disclosure. The restoration parameters, like the detection interval, are determined by the information determination unit 62. The information determination unit 62 includes a learning model M4 for the restoration parameters. The learning model M4 shown in FIG. 19B is intended to infer the restoration parameters. Similar to the learning model M2 described above, the learning model M4 is capable of updating the weights of links connecting nodes in two adjacent layers. During the learning phase of the learning model M4, a compressed image G5, features of the compressed image G5 detected by the scene detection unit 61, and known restoration parameters for restoring the compressed image G5 to a restored image G6 are input to the learning model M4, and the weights of each link are updated so that the corresponding restoration parameters are output from the learning model M4. Multiple compressed images G5 with known restoration parameters corresponding to the features of various subjects are input to the learning model M4, and the weights of the learning model M4 are repeatedly updated.

[0125] By inputting the compressed image G5 and the features of the compressed image of a new subject detected by the scene detection unit 61 to the learning model M4 that has completed the learning stage, optimal restoration parameters can be output from the learning model M4. Furthermore, the information determination unit 62 may have a machine learning unit 50, as in FIG. 14C .

[0126] In this way, in the second embodiment, the detection intervals and restoration parameters of various compressed image features are known and input into the learning models M3 and M4 to allow learning, and the compressed image features detected by the scene detection unit 61 are input as input parameters into the trained learning models M3 and M4, thereby making it possible to easily generate detection intervals and restoration parameters that match the compressed image features.

[0127] (Third Embodiment) The photodetector 10 according to the first and second embodiments of the present disclosure measures the number of photons using a SPAD. In contrast, the photodetector 10 according to a third embodiment of the present disclosure is characterized in that it measures the number of photons using a QIS (Quanta Image Sensor). FIG. 20 is a block diagram showing the configuration of a pixel array unit 11a according to the third embodiment of the present disclosure. The pixel array unit 11a has a plurality of jots (JOTs) 20a. The plurality of jots 20a constitute a QIS. Each of the plurality of jots 20a has a photoelectric conversion element. The jots 20a are pixels with low saturation capacity (e.g., below the diffraction limit).

[0128] 21 is a diagram illustrating the operation of a jot 20a according to a third embodiment of the present disclosure. A photoelectric conversion element in the jot 20a detects photons and generates an electric charge. When the electric charge of the photoelectric conversion element reaches a saturation capacity, the jot 20a outputs a digital signal of 1 or 0 (for example, 1). Because the jot 20a only has a capacity corresponding to a few electrons of photons (two electrons in the example of FIG. 21), photons can be detected by determining whether the capacity of the jot 20a is saturated.

[0129] The Jot 20a can detect photons in the same way as a SPAD without using avalanche multiplication. The Jot 20a can detect photons in a smaller area than a SPAD, which improves resolution. The Jot 20a also consumes less power than a SPAD.

[0130] The jot 20a can adjust the photon detection interval by, for example, adjusting the recharge interval of the photoelectric conversion element, which allows the photodetector 10 according to the third embodiment of the present disclosure to achieve coded exposure as shown in the first and second embodiments of the present disclosure using the QIS.

[0131] (Application Examples) The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot, construction machinery, or agricultural machinery (tractor).

[0132] 22 is a block diagram showing a schematic configuration example of a vehicle control system 7000, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected via a communication network 7010. In the example shown in FIG. 22 , the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an outside-vehicle information detection unit 7400, an inside-vehicle information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these multiple control units may be an in-vehicle communication network conforming to any standard, such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a Local Area Network (LAN), or FlexRay (registered trademark).

[0133] Each control unit includes a microcomputer that performs arithmetic processing according to various programs, a memory unit that stores the programs executed by the microcomputer or parameters used in various calculations, and a drive circuit that drives various controlled devices. Each control unit includes a network I / F for communicating with other control units via a communication network 7010, and a communication I / F for communicating with devices or sensors inside and outside the vehicle via wired or wireless communication. Figure 22 illustrates the functional configuration of the integrated control unit 7600, including a microcomputer 7610, a general-purpose communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, an audio / video output unit 7670, an in-vehicle network I / F 7680, and a memory unit 7690. The other control units also include a microcomputer, a communication I / F, a memory unit, and the like.

[0134] The drivetrain control unit 7100 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 7100 functions as a control device for a drive force generating device for generating drive force for the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating braking force for the vehicle. The drivetrain control unit 7100 may also function as a control device for an ABS (Antilock Brake System) or an ESC (Electronic Stability Control), etc.

[0135] A vehicle state detection unit 7110 is connected to the drivetrain control unit 7100. The vehicle state detection unit 7110 includes at least one of a gyro sensor that detects the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor that detects the acceleration of the vehicle, or a sensor that detects the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine rotation speed, the rotation speed of the wheels, etc. The drivetrain control unit 7100 performs arithmetic processing using signals input from the vehicle state detection unit 7110, and controls the internal combustion engine, the drive motor, the electric power steering device, the brake device, etc.

[0136] The body system control unit 7200 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 7200. The body system control unit 7200 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0137] The battery control unit 7300 controls the secondary battery 7310, which is the power supply source for the drive motor, in accordance with various programs. For example, information such as battery temperature, battery output voltage, or remaining battery capacity is input to the battery control unit 7300 from a battery device equipped with the secondary battery 7310. The battery control unit 7300 performs arithmetic processing using these signals, and controls the temperature regulation of the secondary battery 7310 or a cooling device or the like equipped in the battery device.

[0138] The outside vehicle information detection unit 7400 detects information outside the vehicle equipped with the vehicle control system 7000. For example, at least one of an imaging unit 7410 and an outside vehicle information detection unit 7420 is connected to the outside vehicle information detection unit 7400. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The outside vehicle information detection unit 7420 includes at least one of an environmental sensor for detecting the current weather or climate, or a surrounding information detection sensor for detecting other vehicles, obstacles, pedestrians, etc. around the vehicle equipped with the vehicle control system 7000.

[0139] The environmental sensor may be, for example, at least one of a raindrop sensor that detects rain, a fog sensor that detects fog, a sunshine sensor that detects the degree of sunshine, and a snow sensor that detects snowfall. The surrounding information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device. The imaging unit 7410 and the outside vehicle information detection unit 7420 may each be provided as an independent sensor or device, or may be provided as a device in which multiple sensors or devices are integrated.

[0140] 23 shows an example of the installation positions of the imaging unit 7410 and the vehicle exterior information detection unit 7420. The imaging units 7910, 7912, 7914, 7916, and 7918 are provided, for example, at least one of the front nose, side mirrors, rear bumper, back door, and upper part of the windshield inside the vehicle cabin of the vehicle 7900. The imaging unit 7910 provided on the front nose and the imaging unit 7918 provided on the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 7900. The imaging units 7912 and 7914 provided on the side mirrors mainly acquire images of the sides of the vehicle 7900. The imaging unit 7916 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918 provided on the upper part of the windshield inside the vehicle cabin is mainly used to detect leading vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0141] 23 shows an example of the imaging ranges of the imaging units 7910, 7912, 7914, and 7916. Imaging range a indicates the imaging range of the imaging unit 7910 provided on the front nose, imaging ranges b and c indicate the imaging ranges of the imaging units 7912 and 7914 provided on the side mirrors, respectively, and imaging range d indicates the imaging range of the imaging unit 7916 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 7910, 7912, 7914, and 7916, a bird's-eye view image of the vehicle 7900 viewed from above can be obtained.

[0142] The outside vehicle information detection units 7920, 7922, 7924, 7926, 7928, and 7930 provided on the front, rear, sides, corners, and above the windshield inside the vehicle cabin of the vehicle 7900 may be, for example, ultrasonic sensors or radar devices. The outside vehicle information detection units 7920, 7926, and 7930 provided on the front nose, rear bumper, back door, and above the windshield inside the vehicle cabin of the vehicle 7900 may be, for example, LIDAR devices. These outside vehicle information detection units 7920 to 7930 are mainly used to detect preceding vehicles, pedestrians, obstacles, etc.

[0143] Returning to FIG. 22 , the explanation will be continued. The outside vehicle information detection unit 7400 causes the imaging unit 7410 to capture an image outside the vehicle and receives the captured image data. The outside vehicle information detection unit 7400 also receives detection information from the connected outside vehicle information detection unit 7420. If the outside vehicle information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the outside vehicle information detection unit 7400 emits ultrasonic waves or electromagnetic waves and receives information on the received reflected waves. Based on the received information, the outside vehicle information detection unit 7400 may perform object detection processing or distance detection processing for people, vehicles, obstacles, signs, text on the road, etc. Based on the received information, the outside vehicle information detection unit 7400 may also perform environment recognition processing for recognizing rainfall, fog, road conditions, etc. Based on the received information, the outside vehicle information detection unit 7400 may also calculate the distance to an object outside the vehicle.

[0144] The outside vehicle information detection unit 7400 may also perform image recognition processing or distance detection processing to recognize people, vehicles, obstacles, signs, or characters on the road surface based on the received image data. The outside vehicle information detection unit 7400 may perform processing such as distortion correction or alignment on the received image data, and may also generate an overhead image or a panoramic image by combining image data captured by different image capturing units 7410. The outside vehicle information detection unit 7400 may also perform viewpoint conversion processing using image data captured by different image capturing units 7410.

[0145] The interior information detection unit 7500 detects information inside the vehicle. A driver state detection unit 7510 that detects the driver's state is connected to the interior information detection unit 7500, for example. The driver state detection unit 7510 may include a camera that captures an image of the driver, a biosensor that detects the driver's biometric information, or a microphone that collects sound from within the vehicle cabin. The biosensor is provided, for example, on the seat or steering wheel, and detects the biometric information of a passenger sitting in the seat or the driver gripping the steering wheel. The interior information detection unit 7500 may calculate the driver's level of fatigue or concentration based on the detection information input from the driver state detection unit 7510, or may determine whether the driver is dozing off. The interior information detection unit 7500 may perform processing such as noise canceling on the collected audio signal.

[0146] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 according to various programs. An input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 may be implemented by a device that can be operated by a passenger, such as a touch panel, a button, a microphone, a switch, or a lever. Data obtained by voice recognition of a voice input through a microphone may be input to the integrated control unit 7600. The input unit 7800 may be, for example, a remote control device using infrared or other radio waves, or an externally connected device such as a mobile phone or a personal digital assistant (PDA) that can operate the vehicle control system 7000. The input unit 7800 may be, for example, a camera, in which case the passenger can input information using gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the passenger may be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on information input by the passenger using the input unit 7800 and outputs the input signal to the integrated control unit 7600. Passengers and the like operate this input unit 7800 to input various data to the vehicle control system 7000 and to instruct processing operations.

[0147] The storage unit 7690 may include a ROM (Read Only Memory) that stores various programs executed by the microcomputer, and a RAM (Random Access Memory) that stores various parameters, calculation results, sensor values, etc. The storage unit 7690 may also be realized by a magnetic storage device such as an HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0148] The general-purpose communication I / F 7620 is a general-purpose communication I / F that mediates communication with various devices present in the external environment 7750. The general-purpose communication I / F 7620 may implement a cellular communication protocol such as GSM (Global System of Mobile communications), WiMAX (registered trademark), LTE (Long Term Evolution), or LTE-Advanced (LTE-A), or other wireless communication protocols such as a wireless LAN (also referred to as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). The general-purpose communication I / F 7620 may connect to a device (e.g., an application server or a control server) present on an external network (e.g., the Internet, a cloud network, or an operator-specific network) via, for example, a base station or an access point. In addition, the general-purpose communication I / F 7620 may connect to a terminal located near the vehicle (for example, a terminal of a driver, pedestrian, or store, or an MTC (Machine Type Communication) terminal) using, for example, P2P (Peer To Peer) technology.

[0149] The dedicated communication I / F 7630 is a communication I / F that supports a communication protocol designed for use in vehicles. The dedicated communication I / F 7630 may implement a standard protocol such as WAVE (Wireless Access in Vehicle Environment), which is a combination of a lower layer IEEE 802.11p and an upper layer IEEE 1609, DSRC (Dedicated Short Range Communications), or a cellular communication protocol. The dedicated communication I / F 7630 typically performs V2X communication, which is a concept including one or more of vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0150] The positioning unit 7640 performs positioning by receiving, for example, GNSS signals from GNSS (Global Navigation Satellite System) satellites (for example, GPS signals from GPS (Global Positioning System) satellites), and generates position information including the latitude, longitude, and altitude of the vehicle. Note that the positioning unit 7640 may identify the current position by exchanging signals with a wireless access point, or may obtain position information from a terminal such as a mobile phone, PHS, or smartphone that has a positioning function.

[0151] The beacon receiving unit 7650 receives, for example, radio waves or electromagnetic waves transmitted from radio stations or the like installed on the road, and acquires information such as the current location, congestion, road closures, required travel time, etc. The function of the beacon receiving unit 7650 may be included in the dedicated communication I / F 7630 described above.

[0152] The in-vehicle device I / F 7660 is a communication interface that mediates connections between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 may establish wireless connections using wireless communication protocols such as wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). Furthermore, the in-vehicle device I / F 7660 may establish a wired connection such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), or MHL (Mobile High-Definition Link) via a connection terminal (and a cable, if necessary) not shown. The in-vehicle device 7760 may include, for example, at least one of a mobile device or a wearable device owned by a passenger, or an information device carried into or attached to the vehicle. The in-vehicle device 7760 may also include a navigation device that searches for a route to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0153] The in-vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals in accordance with a predetermined protocol supported by the communication network 7010.

[0154] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 in accordance with various programs based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. For example, the microcomputer 7610 may calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the acquired information inside and outside the vehicle, and output control commands to the drivetrain control unit 7100. For example, the microcomputer 7610 may perform cooperative control aimed at realizing functions of an Advanced Driver Assistance System (ADAS), including vehicle collision avoidance or impact mitigation, following driving based on the following distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc. In addition, the microcomputer 7610 may perform cooperative control for the purpose of autonomous driving, in which the vehicle travels autonomously without relying on driver operation, by controlling a driving force generating device, a steering mechanism, a braking device, etc. based on information acquired about the vehicle's surroundings.

[0155] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people, and create local map information including information about the vicinity of the vehicle's current location, based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. Furthermore, the microcomputer 7610 may predict dangers, such as a vehicle collision, the approach of a pedestrian, or entry into a closed road, based on the acquired information, and generate a warning signal. The warning signal may be, for example, a signal for generating a warning sound or turning on a warning lamp.

[0156] The audio / image output unit 7670 transmits at least one audio and / or image output signal to an output device capable of visually or audibly notifying vehicle occupants or the outside of the vehicle of information. In the example of FIG. 22 , an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are illustrated as output devices. The display unit 7720 may include, for example, at least one of an on-board display and a head-up display. The display unit 7720 may have an AR (Augmented Reality) display function. The output device may also be other devices, such as headphones, a wearable device such as an eyeglass-type display worn by the occupant, a projector, or a lamp. When the output device is a display device, the display device visually displays results obtained by various processes performed by the microcomputer 7610 or information received from other control units in various formats, such as text, images, tables, and graphs. When the output device is an audio output device, the audio output device converts audio signals, such as reproduced audio data or acoustic data, into analog signals and audibly outputs the analog signals.

[0157] In the example shown in FIG. 22 , at least two control units connected via the communication network 7010 may be integrated into a single control unit. Alternatively, each control unit may be composed of multiple control units. Furthermore, the vehicle control system 7000 may include another control unit not shown. In addition, in the above description, some or all of the functions performed by one of the control units may be assigned to another control unit. In other words, as long as information is transmitted and received via the communication network 7010, predetermined arithmetic processing may be performed by one of the control units. Similarly, a sensor or device connected to one of the control units may be connected to another control unit, and multiple control units may transmit and receive detection information to each other via the communication network 7010.

[0158] A computer program for realizing each function of the signal processing device 2 according to this embodiment described with reference to FIG. 15 can be implemented in any control unit or the like. A computer-readable recording medium storing such a computer program can also be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network without using a recording medium. Furthermore, the computer program may be encrypted, modulated, or compressed and distributed via a wired or wireless line such as the Internet, or stored on a recording medium.

[0159] In the vehicle control system 7000 described above, the photodetector 10 according to the present embodiment described with reference to Fig. 1 can be applied to the image capturing unit 7410 of the application example shown in Fig. 22. This enables the image capturing unit 7410 to output higher quality images at high speed.

[0160] The present technology may have the following configurations: (1) A photodetector including a plurality of pixels arranged in a two-dimensional direction; and a signal processing unit that generates an image based on pixel signals output from each of the plurality of pixels, wherein each of the plurality of pixels includes: a photoelectric conversion element that detects incident photons; and a pixel circuit that generates the pixel signal based on whether the photoelectric conversion element detects a photon in each of a plurality of detection intervals that are arranged in order along a time axis, and each of the plurality of detection intervals has an encoded time width. (2) The photodetector according to (1), including an optimization unit that optimizes the time width of each of the plurality of detection intervals. (3) The photodetector according to (2), wherein the optimization unit optimizes the time width of each of the plurality of detection intervals so as to minimize information loss of the pixel signals. (4) The photodetector according to (2), wherein the optimization unit randomly sets the time width of each of the plurality of detection intervals. (5) The photodetector according to any one of (1) to (4), wherein the plurality of detection intervals are arranged without gaps along the time axis. (6) The photodetector according to any one of (1) to (3), wherein at least two detection intervals adjacent to each other in the time axis direction among the plurality of detection intervals are spaced apart in the time axis direction. (7) The photodetector according to any one of (1) to (6), wherein the pixel circuit generates the pixel signal including the number of detection intervals into which photons are incident among the plurality of detection intervals. (8) The photodetector according to any one of (1) to (7), wherein the signal processing unit includes: a compressed image generation unit that generates a compressed image compressed in the spatial direction and the time direction based on the plurality of pixel signals output from the plurality of pixels; and a restored image generation unit that generates a restored image based on the compressed image, the restored image having improved spatiotemporal resolution compared to the compressed image.(9) The light detection device according to (8), comprising: an information storage unit that stores first information for setting a time width of each of the plurality of detection intervals; and an information setting unit that sets second information necessary for generating the decompressed image, wherein the compressed image generation unit generates the compressed image for one frame based on a plurality of the pixel signals for a plurality of frames generated at the plurality of detection intervals set based on the first information, and the decompressed image generation unit generates the decompressed image from the compressed images for the plurality of frames based on the second information. (10) The light detection device according to (9), wherein the information storage unit stores predetermined first information, and the information setting unit sets predetermined second information. (11) The light detection device according to (9), comprising: a scene detection unit that detects a feature of the compressed image, and an information determination unit that determines the first information and the second information based on the feature of the compressed image detected by the scene detection unit. (12) The photodetector according to (11), wherein the information determination unit determines the first information and the second information based on features including at least one of a type of subject and illuminance included in the compressed image. (13) The photodetector according to any one of (1) to (12), wherein the photoelectric conversion element has a SPAD (Single Photon Avalanche Diode). (14) The photodetector according to any one of (1) to (12), wherein the photoelectric conversion element is provided for each jot of a QIS (Quanta Image Sensor). (15) A photodetection system comprising: a photodetection device that generates a plurality of pixel signals by photoelectric conversion; and a signal processing device that generates an image based on the plurality of pixel signals, wherein the photodetection device has a plurality of pixels that are arranged in a two-dimensional direction and generate the plurality of pixel signals, each of the plurality of pixels having: a photoelectric conversion element that detects incident photons; and a pixel circuit that generates the pixel signal based on whether the photoelectric conversion element detects a photon in each of a plurality of detection intervals that are arranged in order along a time axis, and each of the plurality of detection intervals has an encoded time width.(16) The optical detection system according to (15), wherein the optical detection device has a compressed image generation unit that generates a compressed image compressed in spatial and temporal directions based on the plurality of pixel signals, and the signal processing device has a restored image generation unit that generates a restored image based on the compressed image and having improved spatiotemporal resolution compared to the compressed image. (17) The optical detection system according to (16), wherein the signal processing device has an information storage unit that stores first information for setting a time width of each of the plurality of detection intervals, an information setting unit that sets second information necessary for generating the restored image, a scene detection unit that detects features of the compressed image, and an information determination unit that determines the first information and the second information based on the features of the compressed image detected by the scene detection unit. (18) The light detection system according to (16), wherein the light detection device comprises: an information storage unit that stores first information for setting a time width of each of the plurality of detection intervals; an information setting unit that sets second information necessary for generating the restored image; a scene detection unit that detects features of the compressed image; and an information determination unit that determines the first information and the second information based on the features of the compressed image detected by the scene detection unit. (19) The light detection system according to any one of (16) to (18), wherein the signal processing device comprises a signal processing unit that performs predetermined signal processing based on the restored image. (20) The light detection system according to any one of (15) to (19), wherein the light detection device and the signal processing device are configured on different semiconductor chips. (21) An information processing device comprising: a model construction unit that uses image data captured by photoelectric conversion according to the amount of incident light and multiple known detection intervals for photon detection corresponding to the image data as training data, and trains the learning model so that when the image data included in the training data is input to the learning model, the corresponding multiple known detection intervals are output; and an inference unit that inputs image data captured by photoelectric conversion according to the amount of incident light, for which optimal multiple detection intervals are unknown, to the learning model trained by the model construction unit, and outputs the optimal multiple detection intervals from the trained learning model.(22) An information processing device comprising: a model construction unit that uses a compressed image with known restoration parameters for restoring an uncompressed image generated based on photons detected at multiple detection intervals for photon detection and the restoration parameters as training data, and trains a learning model so that when a compressed image with known restoration parameters is input to the learning model, the corresponding restoration parameters are output; and an inference unit that inputs a new compressed image with unknown restoration parameters to the learning model that has been trained by the model construction unit, and outputs the corresponding restoration parameters from the trained learning model. (23) An information processing device comprising: a model construction unit that uses image data captured by photoelectric conversion according to the amount of incident light, multiple known detection intervals for photon detection corresponding to the image data, and features of the image data as training data, and trains the learning model so that when the image data and features of the image data included in the training data are input to the learning model, the corresponding multiple known detection intervals are output; and an inference unit that inputs image data captured by photoelectric conversion according to the amount of incident light and for which optimal multiple detection intervals are unknown and features of the image data to the learning model trained by the model construction unit, and outputs the optimal multiple detection intervals from the trained learning model. (24) An information processing device comprising: a model construction unit that uses a compressed image with known restoration parameters for restoring an image before compression generated based on photons detected at multiple detection intervals for photon detection, the restoration parameters, and features of the compressed image as training data, and trains the learning model so that when the compressed image with known restoration parameters and the features of the compressed image are input to the learning model, the corresponding restoration parameters are output; and an inference unit that inputs a new compressed image with unknown restoration parameters and features of the new compressed image to the learning model already trained by the model construction unit, and outputs the corresponding restoration parameters from the already trained learning model.(25) A program causing a computer to execute: a model construction step of using, as training data, image data captured by photoelectric conversion according to the amount of incident light and a plurality of known detection intervals for photon detection corresponding to the image data, and training the learning model so that when the image data included in the training data is input to the learning model, the corresponding plurality of known detection intervals are output; and an inference step of inputting, to the learning model trained in the model construction step, image data captured by photoelectric conversion according to the amount of incident light and whose optimal plurality of detection intervals are unknown, and outputting the optimal plurality of detection intervals from the trained learning model. (26) A program causing a computer to execute: a model construction step of using, as training data, a compressed image with known restoration parameters for restoring an uncompressed image generated based on photons detected at a plurality of detection intervals for photon detection and the restoration parameters, and training the learning model so that when the compressed image with known restoration parameters is input to the learning model, the corresponding restoration parameters are output; and an inference step of inputting, to the learning model trained by the model construction unit, a new compressed image with unknown restoration parameters, and outputting the corresponding restoration parameters from the trained learning model. (27) A program that causes a computer to execute: a model construction step of using image data captured by photoelectric conversion according to the amount of incident light, multiple known detection intervals for photon detection corresponding to the image data, and features of the image data as training data, and training the learning model so that when the image data and features of the image data included in the training data are input to the learning model, the corresponding multiple known detection intervals are output; and an inference step of inputting image data captured by photoelectric conversion according to the amount of incident light, for which multiple optimal detection intervals are unknown, and features of the image data, to the learning model that has been trained by the model construction unit, and outputting the multiple optimal detection intervals from the trained learning model.(28) A program that causes a computer to execute: a model construction step of training a learning model using a compressed image with known restoration parameters for restoring an uncompressed image generated based on photons detected at multiple detection intervals for photon detection, the restoration parameters, and features of the compressed image as training data, so that when the compressed image with known restoration parameters and the features of the compressed image are input to the learning model, the corresponding restoration parameters are output; and an inference step of inputting a new compressed image with unknown restoration parameters and features of the new compressed image to the learning model that has already been trained by the model construction unit, and outputting the corresponding restoration parameters from the trained learning model.

[0161] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents.

[0162] 1, 1a, 1b Photodetection system, 2, 2a Signal processing device, 10, 10a, 10b Photodetection device, 11, 11a Pixel array unit, 12, 12a Signal processing unit, 13 Control unit, 14 Compressed image generation unit, 15 Restored image generation unit, 16 Image processing unit, 17 Optimization unit, 18 Information storage unit, 18a Register, 19 Information setting unit, 20 Pixel, 20a Jot, 21 Light receiving unit, 22 Counter, 23 Selection switch, 30 Photoelectric conversion element, 31 Inverter, 32 Detection circuit, 41 Light absorption unit, 42 Charge multiplication unit, 43 On-chip lens, 50 Machine learning unit, 51 Model construction unit, 52 Information acquisition unit, 53 Inference unit, 61 Scene detection unit, 62 Information determination unit

Claims

1. A photodetection device comprising: a plurality of pixels arranged in a two-dimensional direction; and a signal processing unit that generates an image based on pixel signals output from each of the plurality of pixels, each of the plurality of pixels having a photoelectric conversion element that detects incident photons, and a pixel circuit that generates the pixel signal based on whether the photoelectric conversion element detects a photon in each of a plurality of detection intervals that are arranged in sequence along a time axis, each of the plurality of detection intervals having an encoded time width.

2. The optical detection device according to claim 1, further comprising an optimization unit that optimizes the time width of each of the plurality of detection intervals.

3. The light detection device according to claim 2, wherein the optimization section optimizes the time width of each of the plurality of detection intervals so as to minimize information loss of the pixel signal.

4. The light detection device according to claim 2, wherein the optimization section randomly sets the time width of each of the plurality of detection intervals.

5. The optical detection device according to claim 1, wherein the plurality of detection intervals are arranged without gaps in the time axis direction.

6. The optical detection device according to claim 1, wherein at least two detection intervals adjacent to each other in the time axis direction among the plurality of detection intervals are spaced apart in the time axis direction.

7. The photodetection device according to claim 1, wherein the pixel circuit generates the pixel signal including the number of detection intervals into which a photon is incident, among the plurality of detection intervals.

8. The optical detection device of claim 1, wherein the signal processing unit comprises: a compressed image generation unit that generates a compressed image compressed in the spatial and temporal directions based on the pixel signals output from the pixels; and a restored image generation unit that generates a restored image based on the compressed image, the restored image having improved spatiotemporal resolution compared to the compressed image.

9. An optical detection device as described in claim 8, comprising: an information storage unit that stores first information for setting the time width of each of the multiple detection intervals; and an information setting unit that sets second information required to generate the restored image, wherein the compressed image generation unit generates one frame of the compressed image based on a plurality of the pixel signals for a plurality of frames generated at the multiple detection intervals set based on the first information, and the restored image generation unit generates the restored image from the compressed images for the plurality of frames based on the second information.

10. The optical detection device according to claim 9, wherein the information storage section stores the first information that is predetermined, and the information setting section sets the second information that is predetermined.

11. The optical detection device as described in claim 9, comprising: a scene detection unit that detects characteristics of the compressed image; and an information determination unit that determines the first information and the second information based on the characteristics of the compressed image detected by the scene detection unit.

12. The light detection device according to claim 11, wherein the information determination unit determines the first information and the second information based on features including at least one of a type of subject and illuminance included in the compressed image.

13. The photodetector according to claim 1, wherein the photoelectric conversion element comprises a SPAD (Single Photon Avalanche Diode).

14. The photodetector according to claim 1, wherein the photoelectric conversion element is provided for each jot of a QIS (Quanta Image Sensor).

15. A photodetection system comprising: a photodetection device that generates a plurality of pixel signals by photoelectric conversion; and a signal processing device that generates an image based on the plurality of pixel signals, wherein the photodetection device has a plurality of pixels that are arranged in a two-dimensional direction and generate the plurality of pixel signals, each of the plurality of pixels having a photoelectric conversion element that detects an incident photon, and a pixel circuit that generates the pixel signal based on whether or not the photoelectric conversion element detects a photon in each of a plurality of detection intervals that are arranged in sequence in the time axis direction, and each of the plurality of detection intervals has an encoded time width.

16. The optical detection system of claim 15, wherein the optical detection device has a compressed image generation unit that generates a compressed image compressed in the spatial and temporal directions based on the multiple pixel signals, and the signal processing device has a restored image generation unit that generates a restored image based on the compressed image, the restored image having improved spatiotemporal resolution compared to the compressed image.

17. The optical detection system of claim 16, wherein the signal processing device has: an information storage unit that stores first information for setting the time width of each of the multiple detection intervals; an information setting unit that sets second information required to generate the restored image; a scene detection unit that detects characteristics of the compressed image; and an information determination unit that determines the first information and the second information based on the characteristics of the compressed image detected by the scene detection unit.

18. The optical detection system of claim 16, wherein the optical detection device comprises: an information storage unit that stores first information for setting the time width of each of the multiple detection intervals; an information setting unit that sets second information required to generate the restored image; a scene detection unit that detects characteristics of the compressed image; and an information determination unit that determines the first information and the second information based on the characteristics of the compressed image detected by the scene detection unit.

19. The optical detection system according to claim 16, wherein the signal processing device has a signal processing section that performs predetermined signal processing based on the restored image.

20. The optical detection system according to claim 15, wherein the optical detection device and the signal processing device are configured on different semiconductor chips.

Images