Light detection device and light detection system

Abstract

[Problem] To improve temporal-spatial resolution and generate an image having high image quality. [Solution] This light detection device is provided with: a plurality of pixels arranged in a two-dimensional direction; and a signal processing unit for generating an image based on the pixel signal output from each of the plurality of pixels. Each of the plurality of pixels includes: a photoelectric conversion element for detecting an incident photon; and a pixel circuit for generating a pixel signal based on whether or not the photoelectric conversion element has detected photons in each of a plurality of detection intervals sequentially arranged in the time axis direction. Each detection interval of the plurality of detection intervals has an encoded time width.

Description

Technical Field

[0001] This disclosure relates to optical detection devices and optical detection systems. Background Technology

[0002] There are known light detection devices that use SPADs (single-photon avalanche diodes), which amplify electrons from an incident photon like an avalanche. Compared to conventional CMOS (complementary metal-oxide-semiconductor) diodes, SPADs can detect even weak light and can detect light at high speeds. As an example of an application of light detection devices using SPADs, a technique has been proposed to extend the dynamic range by adjusting the length of the photon detection period according to the illuminance (see Patent Document 1).

[0003] Reference List

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2023-059522A Summary of the Invention

[0006] Technical issues

[0007] However, Patent Document 1 has the following problems: extending the photon detection period unintentionally leads to a decrease in temporal resolution. Furthermore, for moving subjects, artifacts corresponding to the detection period pattern are generated, which unintentionally degrades the image data.

[0008] In view of this, the present disclosure provides a light detection apparatus and a light detection system that can enhance spatiotemporal resolution and generate high-quality images.

[0009] Technical solutions to technical problems

[0010] To address the aforementioned problems, this disclosure provides a light detection device, comprising: Multiple pixels, arranged in two dimensions, and The signal processing unit generates an image based on the pixel signal output from each pixel out of a plurality of pixels, wherein, Each of the multiple pixels has: Photoelectric conversion element, detecting incident photons, and The pixel circuit generates a pixel signal based on whether a photon has been detected in each of multiple detection time periods arranged sequentially along the time axis by photoelectric conversion elements. Each of the multiple detection periods has an coded time width.

[0011] The optical detection device may also include an optimization unit that optimizes the time width of each detection period among multiple detection periods.

[0012] The optimization unit can optimize the time width of each detection period in multiple detection periods, thereby minimizing the information loss of the pixel signal.

[0013] The optimization department can randomly set the time width of each detection period in multiple detection periods.

[0014] The detection periods in multiple detection periods can be arranged without gaps along the time axis.

[0015] At least some pairs of detection periods, which are included in multiple detection periods and are adjacent to each other along the time axis, can be configured such that the detection periods in each pair of at least some pairs of detection periods are spaced apart from each other along the time axis.

[0016] The pixel circuit can generate pixel signals including the number of detection periods in which photons have been incident, across multiple detection periods.

[0017] The signal processing unit may include: The compressed image generation unit generates a compressed image that has been spatially and temporally compressed based on multiple pixel signals output from multiple pixels, and The image reconstruction generation unit generates a reconstructed image with enhanced spatiotemporal resolution compared to the compressed image, based on the compressed image.

[0018] The optical detection device may also include: The information storage unit stores first information used to set the time width of each detection period among multiple detection periods, and The information setting unit sets the second information required to generate the reconstructed image, wherein... The compressed image generation unit can generate a compressed image of one frame based on multiple pixel signals of multiple frames generated during multiple detection periods set according to the first information, and The reconstructed image generation unit can generate a reconstructed image from multiple frames of compressed images based on the second information.

[0019] The information storage unit can store predetermined first information, and the information setting unit can set predetermined second information.

[0020] The optical detection device may also include: The scene detection unit detects features of compressed images, and The information determination unit determines first information and second information based on features of the compressed image detected by the scene detection unit.

[0021] The information determination unit can determine first information and second information based on features including at least one of the type of subject and illuminance in the compressed image.

[0022] Photoelectric conversion elements can have SPAD (single-photon avalanche diode).

[0023] The photoelectric conversion element can be set for each quantum pixel of the QIS (Quantum Image Sensor).

[0024] Furthermore, this disclosure provides a light detection system, comprising: The light detection device generates multiple pixel signals through photoelectric conversion, and The signal processing device generates an image based on multiple pixel signals, wherein... The light detection device has multiple pixels that generate multiple pixel signals, and these multiple pixels are arranged in two dimensions. Each of the multiple pixels has: Photoelectric conversion element, detecting incident photons, and The pixel circuit generates a pixel signal based on whether a photon has been detected in each of multiple detection time periods arranged sequentially along the time axis by photoelectric conversion elements. Each of the multiple detection periods has an coded time width.

[0025] The optical detection device may include a compressed image generation unit that generates a compressed image that has been spatially and temporally compressed based on multiple pixel signals, and

[0026] The signal processing apparatus may have a reconstructed image generation unit that generates a reconstructed image with enhanced spatiotemporal resolution compared to the compressed image based on the compressed image.

[0027] Signal processing devices may have: The information storage unit stores initial information used to set the time width of each of the multiple detection time periods. The information setting unit sets the second information required to generate the reconstructed image. The scene detection unit detects features of compressed images, and The information determination unit determines first information and second information based on features of the compressed image detected by the scene detection unit.

[0028] Optical detection devices can have: The information storage unit stores initial information used to set the time width of each of the multiple detection time periods. The information setting unit sets the second information required to generate the reconstructed image. The scene detection unit detects features of compressed images, and The information determination unit determines first information and second information based on features of the compressed image detected by the scene detection unit.

[0029] The signal processing apparatus may have a signal processing unit that performs predetermined signal processing based on the reconstructed image.

[0030] Optical detection devices and signal processing devices may include semiconductor chips that are different from each other. Attached Figure Description

[0031] Figure 1 This is a block diagram illustrating the configuration of a light detection apparatus according to a first embodiment of the present disclosure.

[0032] Figure 2A A first example of a stacked layer structure for a photodetector is shown.

[0033] Figure 2B A second example of a stacked layer structure for a light detection device is shown.

[0034] Figure 3 This is a block diagram illustrating the configuration of pixels according to a first embodiment of the present disclosure.

[0035] Figure 4 This is a circuit diagram showing the configuration of the optical receiver according to the first embodiment of the present disclosure.

[0036] Figure 5 This is a cross-sectional view of a photoelectric conversion element according to the first embodiment of the present disclosure.

[0037] Figure 6A It is a timing diagram of each signal of a pixel according to the first embodiment of the present disclosure.

[0038] Figure 6B Photon detection during the measurement period of a pixel is shown according to a first embodiment of this disclosure.

[0039] Figure 7A The characteristics of long and short detection periods are shown in low-light image capture scenarios.

[0040] Figure 7B The characteristics of long and short detection periods are shown in high-light image capture scenarios.

[0041] Figure 8 The measurement period is shown by combining long and short detection periods.

[0042] Figure 9 It is a graph showing the relationship between illuminance and the number of photons detected during each detection period.

[0043] Figure 10A The periodic measurement periods are shown.

[0044] Figure 10B It shows in Figure 10A Artifacts generated during the measurement period.

[0045] Figure 10C yes Figure 10B A magnified image of the artifacts in the image.

[0046] Figure 11 This is a flowchart of image generation performed by the light detection apparatus according to the first embodiment of the present disclosure.

[0047] Figure 12A A first configuration example of a measurement period according to a first embodiment of the present disclosure is shown.

[0048] Figure 12B A second configuration example of a measurement period according to a first embodiment of the present disclosure is shown.

[0049] Figure 13A The subject is shown.

[0050] Figure 13B A compressed image, subjected to space-time compression, is shown according to a first embodiment of the present disclosure.

[0051] Figure 13C A reconstructed image according to a first embodiment of the present disclosure is shown.

[0052] Figure 14A This is a diagram illustrating machine learning for multiple detection periods according to a first embodiment of the present disclosure.

[0053] Figure 14B This is a diagram illustrating machine learning of reconstruction parameters according to a first embodiment of the present disclosure.

[0054] Figure 14C This is a block diagram illustrating the configuration of the machine learning unit according to a first embodiment of the present disclosure.

[0055] Figure 15 This is a block diagram showing a light detection system according to a first embodiment of the present disclosure.

[0056] Figure 16A An image capture operation performed by a light detection device according to a first comparative example is shown.

[0057] Figure 16B Compressed and reconstructed images based on the first comparative example are shown.

[0058] Figure 17A An image capture operation performed by a light detection device according to a second comparative example is shown.

[0059] Figure 17B The measurement period of the light detection device according to the second comparative example is schematically shown.

[0060] Figure 18A This is a block diagram illustrating a first configuration example of a light detection system according to a second embodiment of the present disclosure.

[0061] Figure 18B This is a block diagram illustrating a second configuration example of a light detection system according to a second embodiment of the present disclosure.

[0062] Figure 19A This is a diagram illustrating machine learning during a detection period according to a second embodiment of the present disclosure.

[0063] Figure 19B This is a diagram illustrating machine learning of reconstruction parameters according to the second embodiment of this disclosure.

[0064] Figure 20 This is a block diagram showing the configuration of the pixel array section according to the third embodiment of the present disclosure.

[0065] Figure 21 This is a diagram illustrating the operation performed by a quantum pixel according to a third embodiment of the present disclosure.

[0066] Figure 22 It is a block diagram depicting an example of a schematic configuration of a vehicle control system.

[0067] Figure 23 This is an illustration showing an example of the installation location of the vehicle exterior information detection unit and the imaging unit. Detailed Implementation

[0068] In the following description, embodiments of the optical detection apparatus and optical detection system are explained with reference to the accompanying drawings. Although the following description focuses on the main components of the optical detection apparatus and optical detection system, the optical detection apparatus and optical detection system may have components or functions that are not shown or not described. The following description does not exclude components or functions that are not shown or not described.

[0069] (First Implementation)

[0070] Figure 1 This is a block diagram showing the configuration of a light detection device 10 according to a first embodiment of the present disclosure. The light detection device 10 detects incident light and outputs an image based on the incident light. For example, the light detection device 10 is mounted on a smartphone, digital camera, personal computer, vehicle camera, IoT (Internet of Things) camera, etc. The light detection device 10 includes a pixel array unit 11, a signal processing unit 12, and a control unit 13. As will be mentioned later, Figure 1 Some components of the light detection device 10 shown in the figure can be set in a device different from the light detection device 10 (e.g., a signal processing device mentioned later).

[0071] The pixel array section 11 has a plurality of pixels 20 arranged in a first direction X and a second direction Y. In this specification, [the following will be used to describe the pixels]. Figure 1 The left and right (horizontal) directions are called the first direction X, and... Figure 1 The vertical direction is referred to as the second direction Y. Furthermore, in this specification, a row of pixels 20 arranged in the first direction X is called a pixel row. A column of pixels 20 arranged in the second direction Y is called a pixel column.

[0072] Each pixel 20 has a photoelectric conversion element that detects incident photons and generates a pixel signal based on the photons detected during a predetermined measurement period. For example, the pixel signal includes grayscale information based on the amount of incident light. Alternatively, the pixel signal may include event information representing changes in the amount of incident light. Pixels 20 that output pixel signals including grayscale information are referred to as grayscale pixels, and pixels 20 that output pixel signals including event information are referred to as EVS (Event-Based Vision Sensor) pixels. The pixel array 11 may include a plurality of grayscale pixels or may include a plurality of EVS pixels. Furthermore, the pixel array 11 may include a mixture of grayscale pixels and EVS pixels. Note that in this specification, an example of the pixel array 11 including a plurality of grayscale pixels has been described.

[0073] Each pixel column of the plurality of pixels 20 is connected to a vertical signal line VSL. Each vertical signal line VSL inputs the pixel signals output from all pixels 20 in the corresponding pixel column to the signal processing unit 12.

[0074] The signal processing unit 12 generates an image based on multiple pixel signals output from each pixel in the multiple pixels 20. The signal processing unit 12 includes a compressed image generation unit 14, a reconstructed image generation unit 15, and an image processing unit 16.

[0075] Based on multiple pixel signals, the compressed image generation unit 14 generates a compressed image that has been compressed spatially and temporally. The generated compressed image is an image that includes blurring (hereinafter referred to as blur), and further includes non-uniform blurring. In addition, the compressed image generation unit 14 is capable of generating a single frame of compressed image based on pixel signals from multiple frames.

[0076] Based on multiple compressed images (blurred images) repeatedly generated by the compressed image generation unit 14, the reconstruction image generation unit 15 generates a reconstructed image with enhanced spatiotemporal resolution compared to the compressed images. The reconstruction image generation unit 15 is capable of reconstructing multiple frames of images from a single frame of compressed image generated based on pixel signals from the original multiple frames.

[0077] For example, the processing performed by the image processing unit 16 can be performed by an ISP (Image Signal Processor). The image processing unit 16 performs predetermined image processing (e.g., demosaicing, distortion correction, etc.) on the reconstructed image generated by the reconstructed image generation unit 15. The image processing unit 16 outputs the image-processed image to a downstream application device or the like. As described above, the reconstructed image generation unit 15 and the image processing unit 16 can be provided to a device different from the light detection device 10 (e.g., a signal processing device mentioned later).

[0078] For example, in sync with the vertical synchronization signal, the control unit 13 sequentially selects pixel rows and drives the pixels 20 row by row. In this case, the pixels 20 among the plurality of pixels 20 output pixel signals synchronously row by row. Furthermore, the pixels 20 among the plurality of pixels 20 can be configured to output pixel signals asynchronously at their own timings. The control unit 13 includes a shift register, an arbitrator, etc.

[0079] The control unit 13 includes an optimization unit 17. The optimization unit 17 adjusts the photon detection period to minimize the information loss of the pixel signal when reconstruction based on the compressed image is performed at the reconstructed image generation unit 15 (or downstream application processor, etc.). The specific operations performed by the optimization unit 17 will be described later.

[0080] The light detection device 10 may have an information storage unit 18 that stores information (first information) for allowing the optimization unit 17 to optimally drive the pixel 20, and an information setting unit 19 that sets information (second information) required to generate the reconstructed image. The information storage unit 18 may be provided in the control unit 13. The second information is also called reconstruction parameters.

[0081] For example, the light detection device 10 may include a stacked chip formed by stacking multiple chips. Figure 2A A first example of the stacked layer structure of the light detection device 10 is shown. Figure 2A The light detection device 10 has a two-layer structure, which includes a pixel chip b1 and a logic chip b2 sequentially bonded together. These chips are joined through vias or the like. Note that the pixel chip b1 and the logic chip b2 may be joined via Cu-Cu or bumps instead of vias.

[0082] For example, a plurality of pixels 20 in the pixel array section 11 are disposed in the pixel chip b1. For example, the signal processing section 12 and the control section 13 are disposed in the logic chip b2.

[0083] Figure 2B A second example of the stacked layer structure of the light detection device 10 is shown. Figure 2BThe light detection device 10 has a three-layer structure, comprising a first pixel chip b3, a second pixel chip b4, and a logic chip b2 sequentially bonded together. Some constituent elements of the pixel 20 (e.g., photoelectric conversion elements and their peripheral circuits) are disposed in the first pixel chip b3. The remaining constituent elements of the pixel 20 are disposed in the second pixel chip b4. By disposing some constituent elements of the pixel 20 in the second pixel chip b2, the light detection device 10 can achieve a larger ratio of photoelectric conversion element area to chip area, as well as enhanced sensitivity and chip miniaturization.

[0084] Note that the constituent elements disposed in each chip are not limited to those described above. Furthermore, the photodetector 10 may include four or more stacked chips or may include a single flat chip.

[0085] Figure 3 This is a block diagram illustrating the configuration of pixel 20 according to a first embodiment of the present disclosure. Pixel 20 includes a light receiver 21, a counter 22, and a selection switch 23.

[0086] The light receiver 21 generates a pulse signal PLS in response to the incident photon and provides the pulse signal PLS to the counter 22. The counter 22 measures the number of pulses in the pulse signal PLS. The measured value of the counter 22 is initialized by the reset signal RSTc. Based on the selection signal SEL, the selection switch 23 switches whether to output the pixel signal Vimg based on the measured value of the counter 22. For example, the selection signal SEL and the reset signal RSTc are provided from the control unit 13.

[0087] Figure 4 This is a circuit diagram showing the configuration of the light receiver 21 according to a first embodiment of the present disclosure. The light receiver 21 includes a photoelectric conversion element 30, an inverter 31, a limiting transistor Tr1, a recharge transistor Tr2, and detection transistors Tr3 and Tr4. Figure 4 In the example, transistor Tr1, as well as transistors Tr2 and Tr3, comprise NMOS (N-channel metal-oxide-semiconductor) transistors. Furthermore, transistor Tr4 comprises a PMOS (P-channel metal-oxide-semiconductor) transistor. Note that the conduction type of transistors Tr1 through Tr4 can be changed as needed. Any or all of transistors Tr1 through Tr3 may comprise an NMOS transistor, and transistor Tr4 may comprise a PMOS transistor.

[0088] For example, photoelectric conversion element 30 is a SPAD. Photoelectric conversion element 30 detects photons incident on pixel 20 and generates an electric charge. Either the anode or cathode of photoelectric conversion element 30 ( Figure 4The cathode in the example is connected to the drain of transistor Tr1. Another of the anodes or cathodes of the photoelectric conversion element 30 ( Figure 4 In the example, the anode is connected to a node whose voltage level is lower than the power supply voltage node VDD.

[0089] The limiting transistor Tr1 toggles whether the photoelectric conversion element 30 and the detection node N1 are connected. Detection node N1 is the node connected to the detection circuit 32, which will be mentioned later. The source of the limiting transistor Tr1 is connected to the detection node N1. A limiting signal CLIP is input to the gate of the limiting transistor Tr1. For example, when a low-level limiting signal CLIP is input, the limiting transistor Tr1 is turned on.

[0090] The recharge transistor Tr2 toggles whether the photoelectric conversion element 30 is charged. The drain of the recharge transistor Tr2 is connected to the detection node N1. The source of the recharge transistor Tr2 is connected to the power supply voltage node VDD. A control signal XRST is input to the gate of the recharge transistor Tr2. For example, when a low-level control signal XRST is input, the recharge transistor Tr2 is turned on.

[0091] Detection transistors Tr3 and Tr4 are included in detection circuit 32. Detection circuit 32 outputs a pulse signal PLS based on whether photons have been detected by photoelectric conversion element 30.

[0092] The drain of the detection transistor Tr3 is connected to the output node N2. The source of the detection transistor Tr3 is connected to the power supply node VDD. The detection signal Vdt is input from the detection node N1 to the detection transistor Tr3. When the voltage level of the detection signal Vdt becomes equal to or lower than a predetermined threshold, the detection transistor Tr3 switches to the on state.

[0093] The source of the detection transistor Tr4 is connected to the ground voltage node. The drain of the detection transistor Tr4 is connected to the output node N2. The inverted signal of the control signal XRST is input to the gate of the detection transistor Tr4 via inverter 31. For example, when the inverted signal of the high-level control signal XRST is input, the detection transistor Tr4 switches to the off state.

[0094] Although the detection circuit 32 has an inverter in some cases that inverts the voltage of the output node N2, etc., from Figure 4 The diagram of such inverters is omitted.

[0095] For example, a limiting signal CLIP and a control signal XRST are input from the control unit 13. Figure 4 In the example shown, the two transistors Tr2 and Tr4 are controlled by the control signal XRST. Note that the detection transistor Tr4 can be controlled independently by a signal different from the control signal XRST.

[0096] Figure 5 This is a cross-sectional view of the photoelectric conversion element 30 according to the first embodiment of this disclosure. The photoelectric conversion element 30 has a light absorption section 41 and a charge multiplication section 42. An on-chip lens 43 for focusing incident light is disposed on the light incident surface of the photoelectric conversion element 30. Figure 5 On the top side of the image (in the image). Although in some cases a color filter or similar device that transmits only a specific wavelength component of the incident light is positioned between the photoelectric conversion element 30 and the on-chip lens 43, from the top side of the image (in the image). Figure 5 Illustrations of such color filters are omitted.

[0097] The light-absorbing section 41 absorbs photons incident through the on-chip lens 43 and generates a charge (e.g., an electron e). When the electron e generated by the light-absorbing section 41 is absorbed, the charge-multiplying section 42 accelerates the electron e through a high electric field, causing the electron e to collide with semiconductor atoms and generate new electron e through ionization. In the charge-multiplying section 42, so-called avalanche multiplication occurs, in which the generated electron e further collides with other semiconductor atoms and generates a large current through a chain reaction of collisional ionization. The photoelectric conversion element 30 is able to multiply an electron by avalanche multiplication, thereby detecting light in units of single photons.

[0098] Figure 6A This is a timing diagram of each signal of pixel 20 according to the first embodiment of the present disclosure. Figure 6A The limiting signal CLIP, the control signal XRST, the cathode voltage Vpd of the photoelectric conversion element 30, the detection signal Vdt, and the pulse signal PLS are shown.

[0099] First, at time t1, the limiting signal CLIP transitions to a high level. Consequently, the limiting transistor Tr1 is turned off, and the photoelectric conversion element 30 and the detection node N1 become electrically disconnected. Thus, the photoelectric conversion element 30 awaits the incidence of photons.

[0100] At time t2, the control signal XRST goes high. Consequently, the recharge transistor Tr2 is turned off, and the detection node N1 is disconnected from the power supply voltage node VDD. Furthermore, the detection transistor Tr4 is turned off, and the detection circuit 32 switches to the drive state.

[0101] When the photoelectric conversion element 30 detects photons and generates a large current through avalanche multiplication at time t3, the cathode voltage Vpd decreases significantly.

[0102] At time t4, the limiting signal CLIP transitions to a low level. Because the limiting transistor Tr1 is turned on, the charge of the photoelectric conversion element 30 is transferred to the detection node N1, and the signal level of the detection signal Vdt decreases. Consequently, the detection transistor Tr3 turns on, and a high-level pulse signal PLS is output from the detection circuit 32.

[0103] At time t5, the control signal XRST goes low. This turns on the recharge transistor Tr2, resets the voltage level of detection node N1, and recharges the photoelectric conversion element 30. Furthermore, the detection transistor Tr4 turns on, stops the drive of the detection circuit 32, and the pulse signal PLS goes low.

[0104] At time t6, the photoelectric conversion element 30 completes recharging, and the cathode voltage Vpd rises to the level before photon detection. Furthermore, due to the reset of the voltage level at detection node N1, the signal level of the detection signal Vdt rises, and the detection transistor Tr3 turns off. As a result, the limiting signal CLIP transitions to a high level, and the photoelectric conversion element 30 once again awaits photon incidence.

[0105] As described above, the period when the limiting signal CLIP is at a high level (time t1 to time t4) is the waiting period for the photoelectric conversion element 30 to wait for the incident photon. In addition, the period when the limiting signal CLIP is at a low level includes the output period of the high-level pulse signal PLS (time t4 to time t5) and the recharging period that continues until the photoelectric conversion element 30 is recharged (time t5 to time t6).

[0106] During the waiting period, the photoelectric conversion element 30 can detect one photon. After detecting a photon during the waiting period, the photoelectric conversion element 30 cannot detect new photons until it is recharged.

[0107] If the photoelectric conversion element 30 detects a photon during the waiting period, it outputs a high-level pulse signal PLS during the output period. If the photoelectric conversion element 30 does not detect a photon during the waiting period, it does not output a high-level pulse signal PLS during the output period.

[0108] By being recharged during the recharge period, the photoelectric conversion element 30 is able to detect photons again during the next waiting period.

[0109] In this specification, the waiting period, output period, and recharge period are collectively referred to as the detection period. Although in Figure 6A In the example shown, one photon is detected during each detection period, but each pixel 20 can detect two or more photons during each detection period.

[0110] Figure 6B Photon detection during the measurement period of pixel 20 is shown. In this specification, the period during which the number of photons in a frame is counted is referred to as the measurement period, for example. Figure 6BAs shown, each measurement period includes multiple detection periods. That is, the photoelectric conversion element 30 is capable of detecting multiple photons during the measurement period. Figure 6B The inverted voltage xVpd of the cathode voltage is shown. The inverted voltage xVpd of the cathode voltage indicates whether a photon has been detected during each detection period.

[0111] also, Figure 6B The pulse signal PLS is shown based on whether a photon has been detected during each detection period. When a photon is detected during each detection period, ... Figure 3 The optical receiver 21 outputs a single pulse signal PLS. The optical receiver 21 does not output multiple pulse signals PLS during a single detection period. Furthermore, the optical receiver 21 does not output pulse signals PLS unless a photon is detected during each detection period.

[0112] like Figure 6B As shown, the pulse signal PLS is output as a digital signal with 1 bit of information indicating whether a photon has been detected during each detection period. For example, the pulse signal PLS outputs 1 if a photon is detected, and outputs 0 if no photon is detected.

[0113] Counter 22 increments its count each time a pulse signal PLS is output. By measuring the number of times the pulse signal PLS is output, counter 22 can measure the number of detection periods within the measurement period in which photons have been detected. For example, pixel 20 can measure the brightness of the subject based on the number of photons measured during a predetermined measurement period.

[0114] As described above, the detection circuit 32 and the counter 22 generate a pixel signal Vimg based on the number of detection periods within the measurement period in which photons have been detected. That is, the pixel signal Vimg is generated based on whether a photon has been detected in each of the multiple detection periods arranged sequentially along the time axis by the photoelectric conversion element 30. In this specification, the detection circuit 32 and the counter 22 are collectively referred to as the pixel circuit. The pixel signal Vimg includes information about the number of detection periods within the measurement period in which photons have been detected. In other words, the pixel circuit generates a pixel signal Vimg that includes the number of detection periods in which photons have been incident.

[0115] exist Figure 6B Within the measurement period, detection periods corresponding to exposure periods are arranged without gaps along the time axis. Note that a non-exposure period can be provided between each pair of detection periods. For example, a non-exposure period can be provided through shutter control.

[0116] Pixel 20 can change the length of the detection period by altering the length of the period during which the CLIP signal is high. In the following text, a relatively long detection period will be referred to as a long detection period, and a relatively short detection period will be referred to as a short detection period.

[0117] Since the number of photons incident on the photoelectric conversion element 30 is enormous, the number of incident photons is estimated by counting some of them. This estimation scheme is also known as the CLK-DET (clock detector) scheme. The photodetector 10 can prevent the counter 22's measurement value from overflowing by using the CLK-DET scheme.

[0118] In the CLK-DET scheme, photon counting loss is intentionally generated. One technique for generating photon counting loss uses a long detection period, which is an extended photon detection period. During the long detection period, counting loss can be generated by not detecting photons other than the one initially detected. Another technique for generating photon counting loss uses a short detection period to thin out the photons, which is a shortened photon detection period.

[0119] Figure 7A The characteristics of long and short detection periods are shown in low-light image capture scenarios. Figure 7B The characteristics of long and short detection periods are shown in high-light image capture scenes. Note that... Figure 7A and Figure 7B An example is shown where a non-exposure period is provided between each pair of short detection periods to thin out the photons.

[0120] like Figure 7A As shown, in low-light image capture scenarios with a small number of incident photons, many photons are incident during the non-exposure period, and the incident photons often cannot be detected during the short detection period. Therefore, in low-light image capture scenarios, it is desirable to use a long detection period that is less likely to cause photon detection loss.

[0121] like Figure 7B As shown, in high-illuminance image capture scenarios with a large number of incident photons, photons are detected in almost all detection periods within a long detection period, and it may not be possible to accurately detect periods when photons are not incident. Therefore, in high-illuminance image capture scenarios, it is desirable to use short detection periods that can thin out and detect photons.

[0122] In addition, the optical detection device 10 can combine long detection periods and short detection periods. Figure 8 The measurement period is shown by combining long and short detection periods. Figure 8 During the measurement period, short detection intervals can be used to thin and detect photons in high-illuminance image capture scenes. Furthermore, in... Figure 8 During the measurement period, using a long detection period in low-light image capture scenarios can prevent photon detection loss.

[0123] Figure 8 An example of alternating long and short detection periods is shown. This is not the only example. The measurement period of the CLK-DET scheme may include a period with two or more consecutive long (or short) detection periods, may include randomly arranged long and short detection periods, or may include three or more detection periods. As will be mentioned later, the photodetector 10 according to the first embodiment of this disclosure further optimizes the sequence of multiple detection periods with different time widths in the measurement period of the CLK-DET scheme.

[0124] Figure 9 This is a graph showing the relationship between illuminance and the number of photons detected during each detection period. Figure 9 In the graph, the vertical axis represents the number of photons detected (count), and the horizontal axis represents the illuminance. Furthermore, Figure 9 Curve W1 shows the results of measurements using a long detection period, curve W2 shows the results of measurements using a short detection period, and curve W3 shows the results of measurements using a combination of long and short detection periods. As curve W1 shows, measurements using a long detection period cannot ensure adequate grayscale in high-illuminance areas. As curve W2 shows, measurements using a short detection period cannot ensure adequate grayscale in low-illuminance areas. As curve W3 shows, measurements using a combination of long and short detection periods can ensure adequate grayscale over a wider range of illuminance areas.

[0125] like Figure 9 As indicated by curve W3, the light detection device 10 according to the first embodiment of this disclosure can expand the dynamic range by combining long detection time periods and short detection time periods to provide appropriate grayscale to a wider illumination range.

[0126] like Figure 6B As shown, regardless of the length of the detection period, only one photon can be detected within a single detection period. Therefore, when using long detection periods, the temporal resolution of the generated image is reduced. Figure 1 The compressed image generation unit 14 generates an image based on the measurement value of the counter 22. In the case of generating an image based on the number of photons detected using multiple detection periods with different time widths that are combined as needed, multiple pixel regions with different illumination levels can be considered to generate image data that has been compressed in space-time (hereinafter, compressed image).

[0127] Based on the compressed image described above, the original image can be reconstructed using existing reconstruction algorithms. Note that the reconstructed original image exhibits image degradation due to the combined use of photon detection performed across multiple detection periods with different time widths.

[0128] Figure 10A An example of performing photon detection using short detection periods D1 and long detection periods D2 arranged alternately in the measurement period is shown. It is assumed that the multiple short detection periods D1 included in the measurement period are of the same length, and the multiple long detection periods D2 included in the measurement period are also of the same length.

[0129] Figure 10B It shows the use of Figure 10A The captured image of the disk-shaped rotating body during the measurement period. Figure 10C yes Figure 10B A magnified view of region Zm in the captured image. (See image below.) Figure 10C As shown, when short detection periods D1 and long detection periods D2 are alternately repeated, periodic stripes are visible in the captured image. These stripes are called motion artifacts and are considered a factor in the degradation of image quality in the captured image.

[0130] As described above, in image capture using the CLK-DET scheme, characteristic motion artifacts (hereinafter, blurring) arise depending on the length and arrangement of multiple detection time periods. As will be mentioned later, the light detection apparatus 10 according to a first embodiment of this disclosure is characterized by performing coded exposures in combination using multiple detection time periods having different time widths to control the generation of blurring in the compressed image. Therefore, when performing reconstruction based on the compressed image, it is possible to generate images that do not produce blurring. Figure 10C High-quality reconstructed images to capture motion artifacts.

[0131] Figure 11 This is a flowchart of image generation performed by the light detection apparatus 10 according to the first embodiment of the present disclosure. First, the optimization unit 17 sets a detection period for generating a compressed image (step S1). For example, the detection period is set based on first information stored in the information storage unit 18. The first information includes setting information about the time width of each of the plurality of detection periods. The information storage unit 18 may have pre-stored the predetermined first information. More specifically, the first information may be stored in the information storage unit 18 during the manufacturing stage of the light detection apparatus 10, or it may be stored in the information storage unit 18 before the processing of step S1 is performed. The control of storing the first information in the information storage unit 18 may be performed by the control unit 13 or by an external device different from the light detection apparatus 10.

[0132] Figure 12AA first configuration example of a measurement period according to a first embodiment of the present disclosure is shown. Figure 11 In step S1, multiple detection time periods are determined ( Figure 12A The detection time periods D1, D2, D3, D4, D5, and D6 in the example. Figure 12A In the measurement periods shown, each of the detection periods D1 to D6 is arranged multiple times. Each of the detection periods D1 to D6 has a time width optimized by the optimization unit 17 (i.e., an encoded time width). In this specification, the exposure performed during each of the multiple detection periods encoded by the optimization unit 17 is referred to as encoded exposure. Each of the detection periods is an exposure period, and multiple exposure periods with different time widths are combined to perform image capture.

[0133] The optimization unit 17 optimizes the temporal width of each of the multiple detection periods to minimize the information loss of the pixel signal Vimg. For example, the optimization unit 17 adjusts the temporal width of the multiple detection periods to intentionally introduce non-uniformity into the blur of the compressed image. For example, by generating long detection periods in the sparse parts of the pixel signal Vimg, information loss due to blur can be reduced. As mentioned later, by introducing non-uniformity into the blur of the compressed image, the spatiotemporal resolution can be enhanced when generating the reconstructed image.

[0134] Furthermore, the optimization unit 17 optimizes the arrangement of detection periods D1 to D6. For example, detection periods D1 to D6 are arranged aperiodically (e.g., randomly). The time width and arrangement of detection periods D1 to D6 can be determined empirically or through machine learning.

[0135] exist Figure 12A In the measurement period, multiple detection periods D1 to D6 are arranged without gaps along the time axis. Non-exposure periods can be arranged between each pair of detection periods. Figure 12B A second configuration example of a measurement period according to a first embodiment of the present disclosure is shown. Figure 12B The measurement period and Figure 12A The difference in the measurement time periods is that, Figure 12B The measurement period includes one or more non-exposure periods. That is, in Figure 12B During the measurement period, at least some pairs of detection periods that are adjacent to each other along the time axis are configured such that the detection periods of each pair of at least some pairs of detection periods are spaced apart from each other along the time axis. The time width and arrangement of each non-exposure period can be determined randomly, or empirically or through machine learning.

[0136] Note that the information regarding multiple detection time periods included in the first information is set within the constraints of hardware constraints or design rules. For example, hardware constraints or design rules include the control scheme of the pixel array unit 11 (uniform control of all pixels, control on a pixel row basis, control on a pixel column basis, or control on a pixel basis), the number of types of detection time periods that can be set, the number of each detection time period, etc.

[0137] Next, the information setting unit 19 sets the reconstruction parameters in the reconstructed image generation unit 15 (step S2). Predetermined reconstruction parameters can be input into the information setting unit 19. Specifically, reconstruction parameters that have already been input during the manufacturing of the light detection device 10, etc., can be set; reconstruction parameters generated through machine learning can be set; or reconstruction parameters input from an external device different from the light detection device 10 can be set. Note that the processing order of steps S1 and S2 can be determined as needed, or steps S1 and S2 can be executed simultaneously.

[0138] Next, an image of the subject is captured using multiple pixels 20 in the pixel array unit 11 (step S3). The control unit 13 is based on... Figure 12A The detection periods D1 to D6, as shown, provide a limiting signal CLIP, a control signal XRST, etc., to each pixel 20 and control the operations performed by the pixel 20. Note that the time widths of the multiple detection periods D1 to D6 can be the same across all pixels 20 in the pixel array section 11, or the time widths of the detection periods D1 to D6 can be different between pixels.

[0139] Figure 13A The subject G1 captured in step S3 is shown. For example, subject G1 is... Figure 13A A moving object that moves to the left.

[0140] The compressed image generation unit 14 generates a compressed image based on multiple pixel signals Vimg output by multiple pixels 20 (and pixel circuits) (step S4). Figure 13B An example of the compressed image G2 generated in step S4 is shown. The compressed image G2 is a spatially and temporally compressed image. Blur is produced because the compressed image G2 is generated by repeated exposures during multiple detection periods with different temporal widths. The blurring of the compressed image G2 is generated based on the coded temporal widths of the multiple detection periods D1 to D6. That is, the compressed image generation unit 14 generates the compressed image based on multiple pixel signals Vimg generated during the multiple detection periods set according to the first information.

[0141] exist Figure 11 In step S4, a compressed image of one frame can be generated based on the pixel signals Vimg of multiple frames. This reduces... Figure 1The data rate between the compressed image generation unit 14 and the reconstructed image generation unit 15. In this case, the reconstructed image generation unit 15 generates multiple frames of reconstructed images from one frame of compressed image.

[0142] Next, the image reconstruction unit 15 performs reconstruction based on the compressed image G2 according to the reconstruction parameters set in step S2 and generates a reconstructed image (step S5). Figure 13C The reconstructed image G3, reconstructed in step S5, is shown. The reconstructed image generation unit 15 is capable of reconstructing image G3 with enhanced spatiotemporal resolution by performing reconstruction based on a compressed image with non-uniform blur.

[0143] The image processing unit 16 generates a reconstructed image by performing predetermined image processing on the reconstructed image G3 (step S6). Next, it determines whether to end image capture (step S7). If image capture is to continue, step S3 and subsequent processing are repeated.

[0144] For example, the optimization unit 17 can use machine learning technology to optimize the detection period. Figure 14A This is a conceptual diagram illustrating an example of how the optimization unit 17 optimizes the detection period using machine learning techniques. Figure 14A In this example, the optimization unit 17 has a learning model M1. During the training phase of the learning model M1, image data G4 with multiple known detection periods are used as input parameters to train the learning model M1. For example, similar to a typical image sensor that does not employ the CLK-DET scheme, the image data G4 input to the learning model M1 during the training phase is image data obtained through a typical scheme of image capture performed based on photoelectric conversion of incident light intensity. The learning model M1 includes multiple layers. Each layer has multiple nodes, and nodes between adjacent layers are coupled by links. During the training phase of the learning model M1, one of the multiple image data G4 obtained through image capture of various subjects using a typical scheme and each combination of multiple known detection periods of the image data are used as teaching data, and the weights of each link in the learning model M1 are updated such that when each image data included in the teaching data is input to the learning model M1, the corresponding multiple detection periods are output.

[0145] By inputting image data obtained through typical image capture schemes and with unknown detection time periods into the trained learning model M1, multiple optimal detection time periods can be output from the learning model M1.

[0146] Similarly, for example, the information setting unit 19 can use machine learning techniques to optimize the reconstruction parameters. Figure 14B This is a diagram illustrating machine learning of reconstruction parameters according to a first embodiment of the present disclosure. Figure 14BThe learning model M2 shown is designed to infer reconstruction parameters using a detection period and a compressed image G5 as input parameters. Note that either the detection period or the compressed image G5 can be omitted from the input parameters. The learning model M2 consists of multiple layers. Each layer has multiple nodes, and nodes between adjacent layers are coupled via links. During the training phase of the learning model M2, each combination of the detection period, the compressed image G5, and the known reconstruction parameters used to reconstruct the reconstructed image G6 based on the compressed image G5 is input as teaching data to the learning model M2, and the weights of each link in the learning model M2 are updated such that the known reconstruction parameters are output from the learning model M2. During the training phase, the weight updates of the learning model M2 are repeated using multiple compressed images G5. During the training phase, it is expected that the learning model M2 will be trained by inputting various compressed images G5 with different subject and background types, sizes, brightness, and hues.

[0147] The reconstruction parameters can be inferred by inputting the detection period and compressed image with unknown reconstruction parameters into the trained learning model M2.

[0148] Figure 14C It shows the use of Figure 14B The diagram shows a block diagram of the configuration of a machine learning unit 50 that uses machine learning techniques to infer and reconstruct parameters. For example, the machine learning unit 50 is located in the information setting unit 19. Figure 14C As shown, the machine learning unit 50 includes a model building unit 51, an information acquisition unit 52, and an inference unit 53. The model building unit 51 builds a learning model M2 through machine learning. The information acquisition unit 52 acquires multiple compressed images and multiple detection time periods corresponding to each compressed image. The inference unit 53 inputs the multiple detection time periods and compressed images acquired by the information acquisition unit 52 into the learning model M2, and infers the reconstruction parameters used to reconstruct compressed images with unknown reconstruction parameters.

[0149] Similarly, the machine learning department can use 50 Figure 14A The machine learning technique shown is used to infer multiple optimal detection time periods. In this case, for example, the machine learning unit 50 is set in the optimization unit 17. The model building unit 51 builds a learning model M1, the information acquisition unit 52 acquires image data known for multiple detection time periods, and the inference unit 53 uses the learning model M1 to infer multiple optimal detection time periods.

[0150] Figure 15 This is a block diagram showing a light detection system 1 according to a first embodiment of the present disclosure. The light detection system 1 includes a light detection device (sensor) 10a and a signal processing device (application processor) 2. Figure 15 The light detection device 10a in the middle has the same as Figure 1 The internal configuration of the optical detection device 10 varies. From... Figure 15 The light detection device 10a in the middle is omitted Figure 1 The optical detection device 10 in the photodetector has a signal processing unit 12 internally configured. Specifically, Figure 1 The reconstructed image generation unit 15 and image processing unit 16 of the signal processing unit 12 are not provided in the signal processing unit 12. Figure 15 Instead of the optical detection device 10a in the chip, it is disposed in the signal processing device 2. Note that, for example, the optical detection device 10a and the signal processing device 2 can be disposed as separate chips or can be integrated on a single chip.

[0151] The light detection device 10a outputs a compressed image G5 based on incident light. The light detection device 10a includes a pixel array unit 11, a compressed image generation unit 14, and a control unit 13. The control unit 13 includes a register 18a. Figure 1 Similar to the information storage unit 18 in the middle, register 18a stores the first information related to the detection period.

[0152] The signal processing device 2 generates a reconstructed image G6 based on the compressed image G5 and performs predetermined image processing. The signal processing device 2 includes a reconstructed image generation unit 15, an image processing unit 16, and an information setting unit 19. The image processing unit 16 performs predetermined signal processing based on the reconstructed image. The reconstructed image generation unit 15 and the image processing unit 16 are included in the signal processing unit 12a. In addition to the above-described components, the signal processing device 2 may include an image display unit, an output interface, etc. Thus, the light detection system 1 is able to capture an image of the subject G7.

[0153] For example, the light detection device 10a and the signal processing device 2 may include semiconductor chips that are different from each other. As a more specific implementation example, the light detection device 10a and the signal processing device 2 may both be housed in a single device (e.g., a smartphone). In this case, for example, the light detection device 10a may be an image sensor, and for example, the signal processing device 2 may be an application processor.

[0154] Figure 16A An image capture operation performed by a light detection device according to a first comparative example is illustrated. The light detection device according to the first comparative example and... Figure 1 The difference between the light detection device 10 and the first comparative example is that the light detection device does not perform coded exposure. Figure 16A During the measurement period, the exposure periods are arranged without gaps among multiple exposure periods, each with a specific time width. Figure 16A The measurement period in this process is also called the long exposure period.

[0155] Figure 16B Compressed image G101 and reconstructed image G102 based on the first comparative example are shown. Figure 16BBlur has been generated in the compressed image G101. Since the blur is not controlled by encoding exposure, the blur in the compressed image G101 is generated uniformly. The reconstructed image G102, reconstructed from the compressed image G101, has degraded image quality unintended due to information loss caused by blur. Therefore, the light detection device according to the first comparative example is unable to generate a high-quality image.

[0156] Figure 17A An image capture operation performed by a light detection device according to a second comparative example is illustrated. The light detection device according to the second comparative example provides multiple non-exposure periods during the measurement period via shutter control and encodes the exposure periods. Encoding the exposure periods results in exposures with varying time widths within the exposure periods and can impart non-uniformity to the blurring of the captured image. Figure 17B The measurement time period of the light detection device according to the second comparative example is schematically shown. Figure 17B As shown, during the measurement period according to the second comparative example, the non-exposure period must be arranged between each pair of adjacent exposure periods. Therefore, the amount of light that can be exposed during the measurement period is reduced, and the signal-to-noise ratio (SN) of the reconstructed image is undesirably lower.

[0157] Compared to the light detection apparatus according to the first comparative example, the light detection apparatus 10 according to the first embodiment of this disclosure can control the blurring of the compressed image by encoding the exposure. Therefore, the light detection apparatus 10 can minimize the information loss caused by compression and reconstruction and output a high-quality image. Furthermore, compared to the light detection apparatus according to the second comparative example, the light detection apparatus 10 does not need to provide a non-exposure period between each pair of exposure periods (i.e., each pair of detection periods) and can increase the amount of light exposed during the measurement period. Thus, the light detection apparatus 10 can output a reconstructed image with a higher S / N ratio than in the second comparative example.

[0158] In this way, since the detection time period for the CLK-DET scheme is optimized in the first embodiment of this disclosure, a compressed image including non-uniform blur can be generated, and the spatiotemporal resolution after reconstruction can be enhanced. Furthermore, since all detection time periods are arranged without gaps and no non-exposure time periods are provided, the amount of light exposed can be increased, and the signal-to-noise ratio (S / N) can be enhanced. Additionally, a single frame of compressed image is generated based on pixel signals from multiple frames, and multiple frames of reconstructed images can be generated from the single frame of compressed image. Therefore, the data rate between the compressed image generation unit 14 and the reconstructed image generation unit 15 can be reduced, thereby reducing the impact of noise and lowering hardware costs. Therefore, it becomes easier to integrate into mobile devices such as smartphones. As described above, the light detection apparatus 10 according to the first embodiment of this disclosure can enhance spatiotemporal resolution and generate high-quality images.

[0159] (Second Implementation)

[0160] In the first embodiment of this disclosure, a preset detection period and reconstruction parameters are used. The detection period and reconstruction parameters can be dynamically determined according to the image capture scene. Figure 18A This is a block diagram illustrating a first configuration example of the light detection system 1 according to a second embodiment of the present disclosure. Figure 18A The light detection system 1a shown includes a light detection device 10b. The light detection device 10b includes a scene detection unit 61 and an information determination unit 62.

[0161] Scene detection unit 61 acquires compressed image G5 from compressed image generation unit 14. Scene detection unit 61 detects features of compressed image G5 (e.g., subject type or illumination). Based on the features of compressed image G5 detected by scene detection unit 61, information determination unit 62 determines first information and second information (i.e., detection period and reconstruction parameters). Information determination unit 62 performs the process of determining the detection period in this way and therefore covers... Figure 1 The optimization unit 17 in the image capture unit performs the following functions. The information determination unit 62 inputs first information into register 18a and second information into information setting unit 19. The features of the compressed image G5 detected by the scene detection unit 61 are also referred to as the features of the image capture scene.

[0162] Figure 18B This is a block diagram illustrating a second configuration example of the light detection system 1 according to a second embodiment of the present disclosure. Figure 18B The light detection system 1b shown is Figure 18A The difference between the light detection system 1a and the signal processing device 2a is that the scene detection unit 61 and the information determination unit 62 are disposed in the signal processing device 2a. Note that the scene detection unit 61 and the information determination unit 62 can be applied to... Figure 1 The optical detection device 10 in the middle.

[0163] The information determination unit 62 can determine the detection period or reconstruction parameters through machine learning. Figure 19A This is a diagram illustrating machine learning during a detection period according to a second embodiment of the present disclosure. Figure 19AThe learning model (second learning model) M3 shown is designed to infer the optimal detection time period using features of the compressed image G5 detected by the scene detection unit 61 as input parameters. Similar to the learning model M2 described above, the weights of each link connecting adjacent layers in the learning model M3 can be updated. During the training phase of the learning model M3, features of compressed images with known detection time periods are input into the learning model M3, and the weights of each link are updated to output the corresponding detection time period. Multiple compressed images corresponding to the features of various subject images and with known detection time periods are input into the learning model M3, and the weight updates of the learning model M3 are repeated.

[0164] By inputting the features of the compressed image detected by the scene detection unit 61 in the new subject image into the trained learning model M3, the learning model M3 can output the optimal detection time period.

[0165] Note that, as in Figure 14A In this case, similar to a typical image sensor that does not employ the CLK-DET scheme, image data from a typical scheme obtained through image capture performed based on photoelectric conversion of incident light quantity can be used as input parameters for the learning model M.

[0166] The features of the subject can be the compressed image itself. That is, the learning model M3 can use the compressed image as input parameters to infer the detection period corresponding to the compressed image. In this case, if the amount of data in the compressed image is large, data compression can be further performed in the scene detection unit 61, and the data obtained thereby can be input into the learning model M3.

[0167] Figure 19B This is a diagram illustrating machine learning for reconstructing parameters according to the second embodiment of this disclosure. Similar to the detection period, the reconstructing parameters are determined by the information determination unit 62. The information determination unit 62 has a learning model M4 for the reconstructing parameters. Figure 19B The learning model M4 shown is designed to infer reconstruction parameters. Similar to the learning model M2 described above, the weights of each link connecting adjacent layers in the learning model M4 can be updated. During the training phase of the learning model M4, the compressed image G5, the features of the compressed image G5 detected by the scene detection unit 61, and the known reconstruction parameters used to reconstruct the reconstructed image G6 based on the compressed image G5 are input into the learning model M4, and the weights of each link are updated so that the corresponding reconstruction parameters are output from the learning model M4. Multiple compressed images G5 corresponding to the features of various subjects and whose reconstruction parameters are known are input into the learning model M4, and the weight updates of the learning model M4 are repeated.

[0168] By inputting the compressed image G5 and the features of the compressed image of the new subject detected by the scene detection unit 61 into the trained learning model M4, the learning model M4 can output the optimal reconstruction parameters. Furthermore, as... Figure 14C As shown, the information determination unit 62 may have a machine learning unit 50.

[0169] In this way, in the second embodiment, the learning models M3 and M4 are pre-trained by inputting features of various compressed images with known detection periods and reconstruction parameters, as well as features of the compressed images detected by the scene detection unit 61, into the trained learning models M3 and M4. This allows for the simple generation of detection periods and reconstruction parameters that match the features of the compressed images.

[0170] (Third Implementation)

[0171] The light detection apparatus 10 according to the first and second embodiments of this disclosure uses a SPAD to measure the number of photons. In contrast, the light detection apparatus 10 according to the third embodiment of this disclosure is characterized by using a QIS (quantum image sensor) to measure the number of photons. Figure 20 This is a block diagram illustrating the configuration of a pixel array unit 11a according to a third embodiment of the present disclosure. The pixel array unit 11a has a plurality of quantum pixels (JOTs) 20a. The plurality of quantum pixels 20a are included in a QIS. Each of the plurality of quantum pixels 20a has a photoelectric conversion element. The quantum pixel 20a is a pixel having a low saturation capacity (e.g., below the diffraction limit).

[0172] Figure 21 This is a diagram illustrating the operation performed by a quantum pixel 20a according to a third embodiment of the present disclosure. The photoelectric conversion element in the quantum pixel 20a detects photons and generates an electric charge. When the charge of the photoelectric conversion element has reached its saturation capacity, the quantum pixel 20a outputs a digital signal of 1 or 0 (e.g., 1). Because the quantum pixel 20a has a charge that interacts with only a few electrons (… Figure 21 The two electrons in the example have the same capacity as the photons, so photons can be detected based on whether the capacity of quantum pixel 20a is saturated.

[0173] The quantum pixel 20a can detect photons similarly to SPADs without using avalanche multiplication. Because the quantum pixel 20a can detect photons with an area smaller than that of a SPAD, it offers enhanced resolution. Furthermore, the quantum pixel 20a consumes less power than a SPAD.

[0174] For example, the photon detection period of the quantum pixel 20a can be adjusted by adjusting the recharge cycle of the photoelectric conversion element. Thus, using QIS, the photodetector 10 according to the third embodiment of this disclosure can realize the coded exposure shown in the first and second embodiments of this disclosure.

[0175] (Application Examples)

[0176] The technology disclosed herein can be applied to a variety of products. For example, the technology disclosed herein can be implemented as a device installed on any type of mobile body (such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, aircraft, drones, ships, robots, construction machinery, or agricultural machinery (tractors)).

[0177] Figure 22 This is a block diagram illustrating an example of a schematic configuration of a vehicle control system 7000, which is an example of a mobile body control system to which the technology according to embodiments of this disclosure can be applied. The vehicle control system 7000 includes multiple electronic control units interconnected via a communication network 7010. Figure 22 In the depicted example, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting the multiple control units can be an in-vehicle communication network conforming to any standard, such as a Controller Area Network (CAN), a Local Area Network (LIN), a Local Area Network (LAN), or FlexRay (registered trademark).

[0178] Each control unit includes: a microcomputer that performs arithmetic processing according to various programs; a storage unit that stores programs executed by the microcomputer, parameters for various operations, etc.; and a drive circuit that drives various control target devices. Each control unit also includes: a network interface (I / F) for communicating with other control units via the communication network 7010; and a communication I / F for communicating with devices, sensors, etc., inside and outside the vehicle via wired or wireless communication. Figure 22 The integrated control unit 7600 shown is configured with a microcomputer 7610, a general communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiver 7650, an in-vehicle equipment I / F 7660, a voice / image output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690. Other control units similarly include microcomputers, communication I / Fs, and storage units.

[0179] The drive system control unit 7100 controls the operation of equipment related to the vehicle's drive system according to various programs. For example, the drive system control unit 7100 acts as a control device to control: drive force generating equipment for generating the vehicle's driving force, such as an internal combustion engine or drive motor; drive force transmission mechanisms for transmitting the driving force to the wheels; steering mechanisms for adjusting the vehicle's steering angle; and braking devices for generating the vehicle's braking force. The drive system control unit 7100 may also have the functions of control devices for anti-lock braking systems (ABS), electronic stability control (ESC), etc.

[0180] The drive system control unit 7100 is connected to a vehicle condition detection unit 7110. The vehicle condition detection unit 7110 includes, for example, at least one of the following: a gyroscope sensor for detecting the angular velocity of the vehicle's axial rotational motion, an acceleration sensor for detecting the vehicle's acceleration, and sensors for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, engine speed, or wheel speed. The drive system control unit 7100 performs arithmetic processing using signals input from the vehicle condition detection unit 7110 and controls the internal combustion engine, drive motor, electric power steering system, braking system, etc.

[0181] The body system control unit 7200 controls the operation of various devices installed on the vehicle body according to various programs. For example, the body system control unit 7200 acts as a control device to control: keyless entry systems, smart key systems, power windows, or various lights such as headlights, reversing lights, brake lights, turn signals, fog lights, etc. In this case, radio waves or signals from various switches sent from mobile devices that replace the key can be input to the body system control unit 7200. The body system control unit 7200 receives these input radio waves or signals and controls the vehicle's door locking devices, power windows, lights, etc.

[0182] The battery control unit 7300 controls the secondary battery 7310, which serves as the power source for the drive motor, according to various programs. For example, the battery control unit 7300 is provided with information from the battery device, including the secondary battery 7310, regarding battery temperature, battery output voltage, and remaining battery charge. The battery control unit 7300 uses these signals to perform arithmetic processing and to perform controls such as regulating the temperature of the secondary battery 7310 or controlling cooling devices installed in the battery device.

[0183] The external information detection unit 7400 detects information about the exterior of the vehicle, including the vehicle control system 7000. For example, the external information detection unit 7400 is connected to at least one of the imaging unit 7410 and the external information detection unit 7420. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The external information detection unit 7420 includes, for example, at least one of: an environmental sensor for detecting current atmospheric or weather conditions, and a surrounding information detection sensor for detecting other vehicles, obstacles, pedestrians, etc., around the vehicle, including the vehicle control system 7000.

[0184] Environmental sensors may be, for example, at least one of the following: a raindrop sensor for detecting rain, a fog sensor for detecting fog, a sunlight sensor for detecting sunlight intensity, and a snow sensor for detecting snowfall. Surrounding information detection sensors may be at least one of the following: an ultrasonic sensor, a radar device, and a LIDAR device (light detection and ranging device or laser imaging detection and ranging device). Each of the imaging unit 7410 and the exterior information detection unit 7420 may be configured as an independent sensor or device, or may be configured as a device integrating multiple sensors or devices.

[0185] Figure 23 Examples of mounting positions for the imaging unit 7410 and the exterior information detection unit 7420 are depicted. Imaging units 7910, 7912, 7914, 7916, and 7918 can be installed at at least one of the following locations: the front nose of the vehicle 7900, the side mirrors, the rear bumper, the rear door, and the upper part of the windshield inside the vehicle. The imaging unit 7910 installed at the front nose and the imaging unit 7918 installed at the upper part of the windshield inside the vehicle primarily acquire images of the front of the vehicle 7900. The imaging units 7912 and 7914 installed at the side mirrors primarily acquire images of the sides of the vehicle 7900. The imaging unit 7916 installed at the rear bumper or rear door primarily acquires images of the rear of the vehicle 7900. The imaging unit 7918 installed at the upper part of the windshield inside the vehicle is mainly used to detect vehicles, pedestrians, obstacles, signals, traffic signs, lanes, etc., ahead.

[0186] Incidentally, Figure 23 Examples of the imaging ranges of the various imaging units 7910, 7912, 7914, and 7916 are shown. Imaging range a represents the imaging range of the imaging unit 7910 located at the front nose. Imaging ranges b and c represent the imaging ranges of the imaging units 7912 and 7914 located at the side mirrors, respectively. Imaging range d represents the imaging range of the imaging unit 7916 located at the rear bumper or rear door. For example, a bird's-eye view of the vehicle 7900 viewed from above is obtained by superimposing the image data captured by the imaging units 7910, 7912, 7914, and 7916.

[0187] Exterior information detection units 7920, 7922, 7924, 7926, 7928, and 7930, located at the front, rear, sides, corners, and above the windshield inside the vehicle 7900, can be, for example, ultrasonic sensors or radar devices. Exterior information detection units 7920, 7926, and 7930, located at the front nose, rear bumper, rear door, and above the windshield inside the vehicle 7900, can be, for example, LIDAR devices. These exterior information detection units 7920 to 7930 are primarily used to detect vehicles, pedestrians, obstacles, etc., ahead.

[0188] Back Figure 22 The description will continue. The exterior information detection unit 7400 causes the imaging unit 7410 to image an image of the exterior of the vehicle and receives the image data. Furthermore, the exterior information detection unit 7400 receives detection information from the exterior information detection section 7420 connected to it. If the exterior information detection section 7420 is an ultrasonic sensor, radar device, or LIDAR device, the exterior information detection unit 7400 transmits ultrasonic waves, electromagnetic waves, etc., and receives information about the received reflected waves. Based on the received information, the exterior information detection unit 7400 can perform processing for detecting objects (such as people, vehicles, obstacles, signs, symbols on the road surface, etc.) or processing for the distance to the detected objects. The exterior information detection unit 7400 can perform environmental recognition processing based on the received information, such as identifying rainfall, fog, road conditions, etc. The exterior information detection unit 7400 can calculate the distance to objects outside the vehicle based on the received information.

[0189] Furthermore, based on the received image data, the vehicle exterior information detection unit 7400 can perform image recognition processing to identify people, vehicles, obstacles, signs, road surface symbols, etc., or to detect the distance to objects. The vehicle exterior information detection unit 7400 can perform processing on the received image data, such as distortion correction and alignment, and combine image data captured by multiple different imaging units 7410 to generate a bird's-eye view or panoramic image. The vehicle exterior information detection unit 7400 can use image data captured by imaging units 7410 including different imaging components to perform viewpoint switching processing.

[0190] The in-vehicle information detection unit 7500 detects information about the interior of the vehicle. For example, the in-vehicle information detection unit 7500 is connected to a driver state detection unit 7510 that detects the driver's state. The driver state detection unit 7510 may include a camera that images the driver, a biosensor that detects the driver's biological information, a microphone that collects sounds from inside the vehicle, etc. The biosensor is disposed, for example, on the seat surface, steering wheel, etc., and detects the biological information of passengers sitting in the seat or the driver holding the steering wheel. Based on the detection information input from the driver state detection unit 7510, the in-vehicle information detection unit 7500 can calculate the driver's fatigue level or the driver's level of concentration, or determine whether the driver is dozing off. The in-vehicle information detection unit 7500 can perform processing such as noise cancellation on the audio signals obtained through sound collection.

[0191] The integrated control unit 7600 controls the overall operation within the vehicle control system 7000 according to various programs. The integrated control unit 7600 is connected to the input unit 7800. The input unit 7800 is implemented using a device that allows passengers to input data, such as a touch panel, button, microphone, switch, lever, etc. The integrated control unit 7600 may be provided with data obtained through voice recognition of voice input via a microphone. The input unit 7800 may be, for example, a remote control device using infrared or other radio waves, or an external connection device such as a mobile phone or personal digital assistant (PDA) that supports the operation of the vehicle control system 7000. The input unit 7800 may be, for example, a camera. In this case, passengers can input information through gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the passenger can be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on information input by the passenger using the aforementioned input unit 7800, and outputs the generated input signal to the integrated control unit 7600. Passengers can input various data or give instructions for processing operations to the vehicle control system 7000 through the operation input unit 7800.

[0192] The storage unit 7690 may include a read-only memory (ROM) for storing various programs executed by a microcomputer and a random access memory (RAM) for storing various parameters, operation results, sensor values, etc. Furthermore, the storage unit 7690 may be a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc.

[0193] The Universal Communication I / F 7620 is a widely used communication I / F that mediates communication with various devices existing in the external environment 7750. The Universal Communication I / F 7620 can implement cellular communication protocols such as GSM, WiMAX, LTE, and LTE-A; or other wireless communication protocols such as Wireless LAN (also known as Wi-Fi), Bluetooth, etc. The Universal Communication I / F 7620 can connect, for example, via a base station or access point to devices (e.g., application servers or control servers) existing on external networks (e.g., the Internet, cloud networks, or company-specific networks). Furthermore, the Universal Communication I / F 7620 can connect, for example, using peer-to-peer (P2P) technology to terminals located near the vehicle (e.g., terminals belonging to drivers, pedestrians, or shopkeepers, or machine-type communication (MTC) terminals).

[0194] The Dedicated Communication I / F 7630 is a communication I / F that supports communication protocols developed for vehicle use. The Dedicated Communication I / F 7630 can implement, for example, standard protocols such as Wireless Access in a Vehicle Environment (WAVE) (which is a combination of IEEE 802.11p as the lower layer and IEEE 1609 as the upper layer), Dedicated Short Range Communication (DSRC), or cellular communication protocols. The Dedicated Communication I / F 7630 typically performs V2X communication as a concept including one or more of the following: vehicle-to-vehicle (V2V) communication, road-to-vehicle (V2V) communication, vehicle-to-home (V2V) communication, and pedestrian-to-vehicle (V2V) communication.

[0195] The positioning unit 7640 performs positioning, for example, by receiving Global Navigation Satellite System (GNSS) signals from GNSS satellites (e.g., GPS signals from Global Positioning System (GPS) satellites), and generates location information including the vehicle's latitude, longitude, and altitude. Incidentally, the positioning unit 7640 can identify its current location by exchanging signals with a wireless access point, or by obtaining location information from a terminal (such as a mobile phone with positioning capabilities, a Personal Handheld System (PHS), or a smartphone).

[0196] The beacon receiver 7650 receives, for example, radio waves or electromagnetic waves transmitted from a radio station installed on a road, and thereby obtains information about its current location, congestion, road closure, and estimated time. Incidentally, the functionality of the beacon receiver 7650 can be included in the aforementioned dedicated communication I / F 7630.

[0197] The in-vehicle device I / F 7660 is a communication interface that mediates the connection between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 can establish a wireless connection using wireless communication protocols such as Wireless LAN, Bluetooth (registered trademark), Near Field Communication (NFC), or Wireless Universal Serial Bus (WUSB). Furthermore, the in-vehicle device I / F 7660 can establish a wired connection via connection terminals (and cables, if necessary) not shown in the figures, through Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI (registered trademark)), Mobile High Definition Link (MHL), etc. The in-vehicle devices 7760 may include, for example, at least one of the following: mobile devices and wearable devices owned by passengers, and information devices loaded into or attached to the vehicle. The in-vehicle device 7760 may also include a navigation device for searching routes to any destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0198] The vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The vehicle network I / F 7680 sends and receives signals according to a predetermined protocol supported by the communication network 7010.

[0199] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information obtained via at least one of the following: general communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiving unit 7650, in-vehicle equipment I / F 7660, and in-vehicle network I / F 7680. For example, the microcomputer 7610 can calculate control target values ​​for the drive force generation device, steering mechanism, or braking device based on the obtained information about the vehicle's interior and exterior, and output control commands to the drive system control unit 7100. For example, the microcomputer 7610 can perform cooperative control aimed at realizing functions of an advanced driver assistance system (ADAS), including collision avoidance or impact buffering for the vehicle, following driving based on following distance, speed-maintaining driving, vehicle collision warning, and lane departure warning. Furthermore, the microcomputer 7610 can control the drive force generation device, steering mechanism, braking device, etc., based on the information obtained about the vehicle's surrounding environment, thereby performing cooperative control intended for autonomous driving, which enables the vehicle to drive automatically without relying on the driver's operation.

[0200] The microcomputer 7610 can generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people based on information obtained via at least one of the following: general communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiving unit 7650, in-vehicle equipment I / F 7660, and in-vehicle network I / F 7680. It also generates local map information including information about the surrounding environment and the vehicle's current location. Furthermore, the microcomputer 7610 can predict hazards such as vehicle collisions, pedestrian approach, and entry into closed roads based on the obtained information and generate alarm signals. These alarm signals can be, for example, signals used to generate alarm sounds or illuminate warning lights.

[0201] The sound / image output unit 7670 sends an output signal of at least one of sound and image to an output device capable of visually or audibly notifying passengers of the vehicle or the outside of the vehicle. Figure 22 In this example, an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are shown as output devices. The display unit 7720 may include, for example, at least one of a vehicle-mounted display and a head-up display. The display unit 7720 may have augmented reality (AR) display functionality. The output device may be other than these devices, and may be another device such as headphones, wearable devices such as glasses displays worn by passengers, projectors, lamps, etc. When the output device is a display device, the display device visually displays the results obtained from various processes performed by the microcomputer 7610, or displays information received from another control unit in various forms (such as text, images, tables, graphs, etc.). Furthermore, when the output device is an audio output device, the audio output device converts an audio signal consisting of played audio data or sound data into an analog signal and outputs the analog signal audibly.

[0202] Incidentally, in Figure 22 In the depicted example, at least two control units connected to each other via communication network 7010 can be integrated into a single control unit. Alternatively, each individual control unit may include multiple control units. Furthermore, the vehicle control system 7000 may include another control unit not depicted in the figures. Additionally, some or all of the functions performed by one control unit in the above-described control units can be assigned to another control unit. That is, predetermined arithmetic processing can be performed by any one control unit, as long as information is sent and received via communication network 7010. Similarly, sensors or devices connected to one control unit can be connected to another control unit, and multiple control units can send and receive detection information to each other via communication network 7010.

[0203] Note that this is used to implement [the system / program]. Figure 15 The computer program for each function of the signal processing apparatus 2 described in this embodiment can be implemented on any control unit or the like. Furthermore, a computer-readable recording medium storing such a computer program can also be provided. For example, the recording medium is a magnetic disk, optical disk, magneto-optical disk, flash memory, etc. Furthermore, for example, the aforementioned computer program can be distributed via a network without using a recording medium. Moreover, the aforementioned computer program, which is in an encrypted, modulated, or compressed state, can be distributed via cable lines or wireless lines such as the Internet, or it can be placed in a recording medium and distributed.

[0204] In the vehicle control system 7000 described above, according to the user Figure 1 The light detection device 10 described in this embodiment can be applied to... Figure 22 The imaging unit 7410 shown in the example application is an example of this. Therefore, the imaging unit 7410 is capable of outputting high-quality images at high speed.

[0205] Note that this technology can have the following configurations.

[0206] (1) A light detection device, comprising: Multiple pixels, arranged in two dimensions, and The signal processing unit generates an image based on the pixel signal output from each of the multiple pixels, wherein... Each of the multiple pixels has: Photoelectric conversion element, detecting incident photons, and The pixel circuit generates a pixel signal based on whether a photon has been detected in each of multiple detection time periods arranged sequentially along the time axis by photoelectric conversion elements. Each of the multiple detection periods has an coded time width.

[0207] (2) The light detection device according to (1) further includes: The optimization department optimizes the time width of each detection period within multiple detection periods.

[0208] (3) According to the optical detection device of (2), wherein, The optimization unit optimizes the time width of each detection period in multiple detection periods to minimize the information loss of the pixel signal.

[0209] (4) According to the optical detection device of (2), wherein, The optimization department randomly sets the time width of each of the multiple detection periods.

[0210] (5) A light detection device according to any one of (1) to (4), wherein, The detection periods in the multiple detection periods are arranged without gaps along the time axis.

[0211] (6) A light detection device according to any one of (1) to (3), wherein, At least some pairs of detection periods, which are included in multiple detection periods and are adjacent to each other along the time axis, are configured such that each pair of detection periods in the at least some pairs of detection periods is spaced apart from each other along the time axis.

[0212] (7) A light detection device according to any one of (1) to (6), wherein, The pixel circuit generates pixel signals including the number of detection periods in which photons have been incident, across multiple detection periods.

[0213] (8) A light detection device according to any one of (1) to (7), wherein, The signal processing unit includes: The compressed image generation unit generates a compressed image that has been spatially and temporally compressed based on multiple pixel signals output from multiple pixels, and The image reconstruction generation unit generates a reconstructed image with enhanced spatiotemporal resolution compared to the compressed image, based on the compressed image.

[0214] (9) The light detection device according to (8) further includes: The information storage unit stores first information used to set the time width of each detection period among multiple detection periods, and The information setting unit sets the second information required to generate the reconstructed image, wherein... The compressed image generation unit generates a compressed image of one frame based on multiple pixel signals of multiple frames generated during multiple detection periods set according to the first information, and The reconstructed image generation unit generates a reconstructed image from the compressed image of multiple frames based on the second information.

[0215] (10) The light detection device according to (9), wherein, The information storage unit stores predetermined first information, and The information setting department sets a predetermined second piece of information.

[0216] (11) The light detection device according to (9) further includes: The scene detection unit detects features of compressed images, and The information determination unit determines first information and second information based on features of the compressed image detected by the scene detection unit.

[0217] (12) The optical detection device according to (11), wherein, The information determination unit determines first information and second information based on features including at least one of the type of subject and illuminance in the compressed image.

[0218] (13) A light detection device according to any one of (1) to (12), wherein, The photoelectric conversion element has a SPAD (single-photon avalanche diode).

[0219] (14) A light detection device according to any one of (1) to (12), wherein, The photoelectric conversion element is set for each quantum pixel of the QIS (Quantum Image Sensor).

[0220] (15) A light detection system, comprising: The light detection device generates multiple pixel signals through photoelectric conversion, and The signal processing device generates an image based on multiple pixel signals, wherein... The light detection device has multiple pixels that generate multiple pixel signals, and these multiple pixels are arranged in two dimensions. Each of the multiple pixels has: Photoelectric conversion element, detecting incident photons, and The pixel circuit generates a pixel signal based on whether a photon has been detected in each of multiple detection time periods arranged sequentially along the time axis by photoelectric conversion elements. Each of the multiple detection periods has an coded time width.

[0221] (16) According to the optical detection system of (15), wherein, The optical detection device has a compressed image generation unit that generates a compressed image that has been spatially and temporally compressed based on multiple pixel signals, and The signal processing apparatus has a reconstructed image generation unit that generates a reconstructed image with enhanced spatiotemporal resolution compared to the compressed image based on the compressed image.

[0222] (17) According to the optical detection system of (16), wherein, The signal processing device has: The information storage unit stores initial information used to set the time width of each of the multiple detection time periods. The information setting unit sets the second information required to generate the reconstructed image. The scene detection unit detects features of compressed images, and The information determination unit determines first information and second information based on features of the compressed image detected by the scene detection unit.

[0223] (18) According to the optical detection system of (16), wherein, The optical detection device has: The information storage unit stores initial information used to set the time width of each of the multiple detection time periods. The information setting unit sets the second information required to generate the reconstructed image. The scene detection unit detects features of compressed images, and The information determination unit determines first information and second information based on features of the compressed image detected by the scene detection unit.

[0224] (19) A light detection system according to any one of (16) to (18), wherein, The signal processing apparatus has a signal processing unit that performs predetermined signal processing based on the reconstructed image.

[0225] (20) A light detection system according to any one of (15) to (19), wherein, The optical detection device and the signal processing device consist of semiconductor chips that are different from each other.

[0226] (21) An information processing apparatus, comprising: Model Construction Department The model building section uses image data obtained through image capture performed based on photoelectric conversion of incident light amount, along with multiple known detection time periods corresponding to this image data for photon detection, as teaching data. The model construction process trains a learning model so that when image data included in the teaching data is input into the learning model, it outputs the corresponding multiple known detection time periods, and reasoning department, The inference unit inputs image data, obtained through image capture performed based on photoelectric conversion of incident light intensity and for which multiple optimal detection time periods are unknown, into a learning model trained by the model building unit. This inference unit enables the trained learning model to output multiple optimal detection time periods.

[0227] (22) An information processing apparatus, comprising: Model Construction Department The model construction unit uses compressed images and reconstruction parameters as teaching data. The reconstruction parameters of the compressed images are known and are used to reconstruct uncompressed images based on photons detected during multiple detection periods for photon detection. The model construction process trains a learning model so that when a compressed image with known reconstruction parameters is input into the learning model, it outputs the corresponding reconstruction parameters, and... reasoning department, The inference unit inputs the newly compressed image with unknown reconstructed parameters into the learning model that has been trained by the model building unit, and The inference unit causes the trained learning model to output the corresponding reconstruction parameters.

[0228] (23) An information processing apparatus, comprising: Model Construction Department The model building section uses image data obtained through image capture performed based on photoelectric conversion of incident light amount, multiple known detection periods for photon detection corresponding to the image data, and features of the image data as teaching data, and The model construction and training process trains a learning model so that when image data and its features, included in the teaching data, are input into the learning model, it outputs multiple known detection time periods, and reasoning department, The inference unit inputs image data obtained through image capture performed based on photoelectric conversion of incident light intensity, where multiple optimal detection time periods are unknown, along with the features of this image data, into a learning model trained by the model building unit. This inference unit enables the trained learning model to output multiple optimal detection time periods.

[0229] (24) An information processing apparatus, comprising: Model Construction Department The model construction unit uses compressed images, reconstruction parameters, and features of the compressed images as teaching data. The reconstruction parameters of the compressed images are known and are used to reconstruct uncompressed images based on photons detected during multiple detection periods for photon detection. The model construction process trains a learning model so that when a compressed image with known reconstruction parameters and its features are input into the learning model, the corresponding reconstruction parameters are output. reasoning department, The inference unit inputs the reconstructed new compressed image with unknown parameters and the features of the new compressed image into the learning model trained by the model building unit, and The inference unit causes the trained learning model to output the corresponding reconstruction parameters.

[0230] (25) A program that causes a computer to perform a process, the process comprising: Execution model building includes: Image data obtained through image capture performed based on the photoelectric conversion of incident light intensity, along with multiple known detection time periods corresponding to this image data for photon detection, are used as teaching data. The learning model is trained so that when image data included in the teaching data is input into the learning model, it outputs the corresponding multiple known detection time periods, and Performing reasoning includes: Image data, obtained through image capture performed based on photoelectric conversion of incident light intensity and for which multiple optimal detection timeframes are unknown, is input into a learning model trained by performing model construction. This allows the trained learning model to output multiple optimal detection time periods.

[0231] (26) A program that causes a computer to perform a process, the process comprising: Execution model building includes: Using a compressed image and reconstruction parameters as teaching data, the reconstruction parameters of the compressed image are known and are used to reconstruct an uncompressed image based on photons detected in multiple detection periods for photon detection. The learning model is trained so that when a compressed image with known reconstruction parameters is input into the learning model, it outputs the corresponding reconstruction parameters, and Perform inference, including: The newly compressed image with unknown reconstruction parameters is input into the learning model that has been trained by the model building department, and This causes the trained learning model to output the corresponding reconstruction parameters.

[0232] (27) A program that causes a computer to perform a process, the process comprising: Execution model building includes: Image data obtained through image capture performed based on photoelectric conversion of incident light intensity, multiple known detection periods corresponding to this image data for photon detection, and features of this image data are used as teaching data. The learning model is trained so that when image data included in the teaching data and the features of that image data are input into the learning model, it outputs the corresponding multiple known detection time periods, and Perform inference, including: Image data acquired through image capture performed based on photoelectric conversion of incident light intensity, with multiple optimal detection time periods unknown, along with the features of this image data, are input into a learning model trained by the model building unit. This allows the trained learning model to output multiple optimal detection time periods.

[0233] (28) A program that causes a computer to perform a process, the process comprising: Execution model building includes: Using a compressed image, reconstruction parameters, and features of the compressed image as teaching data, the reconstruction parameters of the compressed image are known and used to reconstruct an uncompressed image based on photons detected during multiple detection periods for photon detection. The learning model is trained so that when a compressed image with known reconstruction parameters and its features are input into the learning model, it outputs the corresponding reconstruction parameters, and... Perform inference, including: The newly compressed image with unknown reconstruction parameters and its features are input into the learning model already trained by the model building unit. This causes the trained learning model to output the corresponding reconstruction parameters.

[0234] This disclosure is not limited to the various embodiments described above, but also incorporates various modifications that can be conceived by those skilled in the art, and the advantages of this disclosure are not limited to the above. That is, various additions, changes, and partial deletions can be made without departing from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents.

[0235] Reference number list

[0236] 1. 1a, 1b Optical Detection System

[0237] 2, 2a Signal processing device

[0238] 10, 10a, 10b Optical Detection Devices

[0239] 11, 11a Pixel Array Section

[0240] 12, 12a Signal Processing Unit

[0241] 13 Control Department

[0242] 14 Compressed Image Generation Unit

[0243] 15. Image Reconstruction Generation Unit

[0244] 16 Image Processing Department

[0245] 17 Optimization Department

[0246] 18. Information Storage Department

[0247] 18a register

[0248] 19 Information Setting Department

[0249] 20 pixels

[0250] 20a Quantum Pixel

[0251] 21 Optical Receiver

[0252] 22-counter

[0253] 23 Selector Switch

[0254] 30 Photoelectric conversion elements

[0255] 31 Inverter

[0256] 32 Detection Circuit

[0257] 41 Light Absorption Section

[0258] 42. Charge multiplication section

[0259] 43 On-plate lenses

[0260] 50 Machine Learning Department

[0261] 51 Model Construction Department

[0262] 52 Information Acquisition Department

[0263] 53 Reasoning Department

[0264] 61 Scene Detection Department

[0265] 62. Information Determination Department.

Claims

1. A light detection device, comprising: Multiple pixels, arranged in two dimensions, and The signal processing unit generates an image based on the pixel signal output from each of the plurality of pixels, wherein, Each of the plurality of pixels has: Photoelectric conversion element, detecting incident photons, and The pixel circuit generates the pixel signal based on whether a photon has been detected in each of a plurality of detection time periods arranged sequentially along the time axis by the photoelectric conversion element. Each of the plurality of detection periods has an coded time width.

2. The optical detection device according to claim 1, further comprising: The optimization department optimizes the time width of each of the multiple detection time periods.

3. The optical detection device according to claim 2, wherein, The optimization unit optimizes the time width of each of the multiple detection time periods to minimize the information loss of the pixel signal.

4. The optical detection device according to claim 2, wherein, The optimization unit randomly sets the time width of each of the multiple detection time periods.

5. The optical detection device according to claim 1, wherein, The detection periods among the multiple detection periods are arranged without gaps along the time axis.

6. The optical detection device according to claim 1, wherein, At least some pairs of detection periods, which are included in the plurality of detection periods and are adjacent to each other along the time axis, are configured such that the detection periods in each pair of the at least some pairs of detection periods are spaced apart from each other along the time axis.

7. The optical detection device according to claim 1, wherein, The pixel circuit generates a pixel signal comprising the number of detection periods in which photons have been incident.

8. The optical detection device according to claim 1, wherein, The signal processing unit includes: The compressed image generation unit generates a compressed image that has been spatially and temporally compressed based on multiple pixel signals output from the multiple pixels. The reconstructed image generation unit generates a reconstructed image with enhanced spatiotemporal resolution compared to the compressed image, based on the compressed image.

9. The optical detection device according to claim 8, further comprising: The information storage unit stores first information for setting the time width of each of the plurality of detection time periods, and The information setting unit sets the second information required to generate the reconstructed image, wherein... The compressed image generation unit generates a compressed image of one frame based on the pixel signals of multiple frames generated during the multiple detection time periods set according to the first information, and The reconstructed image generation unit generates the reconstructed image from the compressed image of multiple frames based on the second information.

10. The optical detection device according to claim 9, wherein, The information storage unit stores the predetermined first information, and The information setting unit sets the second information that has been predetermined.

11. The optical detection device according to claim 9, further comprising: The scene detection unit detects the features of the compressed image, and The information determination unit determines the first information and the second information based on the features of the compressed image detected by the scene detection unit.

12. The optical detection device according to claim 11, wherein, The information determination unit determines the first information and the second information based on features including at least one of the type of subject and illumination in the compressed image.

13. The optical detection device according to claim 1, wherein, The photoelectric conversion element has a SPAD (single-photon avalanche diode).

14. The optical detection device according to claim 1, wherein, The photoelectric conversion element is configured for each quantum pixel of the QIS (Quantum Image Sensor).

15. A light detection system, comprising: The light detection device generates multiple pixel signals through photoelectric conversion, and A signal processing device generates an image based on the plurality of pixel signals, wherein, The light detection device has multiple pixels that generate the multiple pixel signals, and the multiple pixels are arranged in a two-dimensional manner. Each of the plurality of pixels has: Photoelectric conversion element, detecting incident photons, and The pixel circuit generates the pixel signal based on whether a photon has been detected in each of a plurality of detection time periods arranged sequentially along the time axis by the photoelectric conversion element. Each of the plurality of detection periods has an coded time width.

16. The optical detection system according to claim 15, wherein, The optical detection device includes a compressed image generation unit, which generates a compressed image that has been spatially and temporally compressed based on the plurality of pixel signals. The signal processing apparatus includes a reconstructed image generation unit that generates a reconstructed image with enhanced spatiotemporal resolution compared to the compressed image based on the compressed image.

17. The optical detection system according to claim 16, wherein, The signal processing device has: The information storage unit stores first information for setting the time width of each of the plurality of detection time periods. The information setting unit sets the second information required to generate the reconstructed image. The scene detection unit detects the features of the compressed image, and The information determination unit determines the first information and the second information based on the features of the compressed image detected by the scene detection unit.

18. The optical detection system according to claim 16, wherein, The optical detection device has the following features: The information storage unit stores first information for setting the time width of each of the plurality of detection time periods. The information setting unit sets the second information required to generate the reconstructed image. The scene detection unit detects the features of the compressed image, and The information determination unit determines the first information and the second information based on the features of the compressed image detected by the scene detection unit.

19. The optical detection system according to claim 16, wherein, The signal processing apparatus has a signal processing unit that performs predetermined signal processing based on the reconstructed image.

20. The optical detection system according to claim 15, wherein, The optical detection device and the signal processing device each comprise semiconductor chips that are different from each other.

Images