Light detection device and method for controlling light detection device

Abstract

Provided is a light detection device using a neural network circuit, in which device performance is improved. The sensor reads out sensor data in which a plurality of pixel data items are arranged from the pixel array unit. The neural network circuit processes the sensor data based on a neural network model, and outputs line data in which a plurality of processing results are arranged. The readout control unit generates a readout control signal specifying a pixel group to be read out within the pixel array unit based on the line data.

Description

Technical Field

[0001] The present technology relates to an optical detection device. Specifically, the present technology relates to an optical detection device using a neural network model and a control method for the optical detection device. Background Art

[0002] In recent years, signal processing using a neural network model has been increasingly expanded in functions and applications. For example, a device has been proposed in which each of a plurality of pixels and neural network circuits is divided into a plurality of groups, and pixel information of each image group is input to a corresponding neural network circuit group (see, for example, Patent Document 1).

[0003] Citation List

[0004] Patent Document

[0005] Patent Document 1: Japanese translation of PCT international application, Publication No. 2022-525794 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] In the above conventional technology, pixel information of each image group is input to a corresponding neural network circuit group, thereby accelerating processing. However, in the above device, the higher the output rate of the pixels, the greater the power consumption and processing delay of the subsequent circuits. Therefore, the output rate becomes a bottleneck, and further performance improvement becomes difficult.

[0008] The present invention has been made in view of this situation, and an object thereof is to improve the performance of an optical detection device using a neural network circuit.

[0009] Solution to the Problem

[0010] The present technology has been made to solve the above problems, and a first aspect of the present technology is an optical detection device and a control method for the optical detection device. The optical monitoring device includes: a sensor that reads sensor data from a pixel array unit in which a plurality of pixel data are arranged; a neural network circuit that processes the sensor data based on a neural network model and outputs line data in which a plurality of processing results are arranged; and a readout control unit that generates a readout control signal based on the line data, the readout control signal indicating a readout target pixel group in the pixel array unit. This has the effect of suppressing an increase in processing delay and power consumption of the optical detection device.

[0011] In addition, in the first aspect, the line data may include first line data and second line data. The neural network circuit may output the first line data and the second line data in parallel, and the read control unit compares the first line data with the second line data and generates a read control signal based on the comparison result. This brings the effect of controlling reading in units of rows and columns.

[0012] Furthermore, in the first aspect, the neural network model may include a spiking neural network model. Each of the first line data and the second line data may include a plurality of bit strings, and each of the plurality of bit strings may include a plurality of bits representing the detection result of a pulse in chronological order. This brings the effect of controlling reading based on the detection result of the pulse.

[0013] Moreover, in the first aspect, the neural network model may include a spiking neural network model, and each of the first line data and the second line data may include a plurality of bit strings representing the state values of the membrane potential in chronological order. This brings the effect of controlling reading based on the state values of the membrane potential.

[0014] In addition, in the first aspect, a conversion unit may be further included. The conversion unit converts at least one of the identification information and the sensor data, and provides the converted identification information and / or sensor data to the neural network circuit. The sensor may output the identification information together with the sensor data. This brings the effect of controlling reading based on the identification information and the sensor data.

[0015] Moreover, in the first aspect, a first first-in-first-out (FIFO) memory for holding sensor data in a first-in-first-out manner and a second FIFO memory for holding the read control signal in a first-in-first-out manner may be further included. The neural network circuit may read the sensor data from the first FIFO memory, and the sensor may read the read control signal from the second FIFO memory. This brings the effect of buffering the read control signal and the sensor data.

[0016] Furthermore, in the first aspect, a first first-in-first-out (FIFO) memory for holding sensor data in a first-in-first-out manner and a second FIFO memory for holding the line data in a first-in-first-out manner may be further included. The neural network circuit may read the sensor data from the first FIFO memory, and the read control unit may read the line data from the second FIFO memory. This brings the effect of buffering the sensor data and the line data.

[0017] In addition, in the first aspect, a digital processing unit and a first format processing unit may be further included. The digital processing unit reads the sensor data from the first FIFO memory and processes the sensor data. The first format processing unit generates a communication frame for storing the sensor data. This brings the effect of outputting the sensor data to the outside.

[0018] In addition, in the first aspect, a second format processing unit for generating a communication frame storing line data may be further included, and the readout control unit may output the line data to the second format processing unit. This has the effect of outputting the line data to the outside.

[0019] Furthermore, in the first aspect, the sensor may include an event-based vision sensor (EVS). This has the effect of controlling the reading of the EVS.

[0020] Moreover, in the first aspect, the sensor may include a photon measurement circuit for counting photons. This has the effect of controlling the reading of the photon counting circuit.

[0021] In addition, in the first aspect, the sensor may include a CMOS image sensor (CIS). This has the effect of controlling the reading of the CIS.

[0022] Furthermore, in the first aspect, the sensor, the neural network circuit, and the readout control unit may be dispersedly arranged on a plurality of stacked chips. This has the effect of reducing the circuit scale of each chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a block diagram showing a configuration example of an optical detection device according to a first embodiment of the present technology.

[0024] Figure 2 is a block diagram showing a configuration example of a sensor chip according to a first embodiment of the present technology.

[0025] Figure 3 is a block diagram showing a configuration example of an EVS according to a first embodiment of the present technology.

[0026] Figure 4 is a circuit diagram showing a configuration example of a pixel according to a first embodiment of the present technology.

[0027] Figure 5 is a block diagram showing a configuration example of a spiking neural network (SNN) processor according to a first embodiment of the present technology.

[0028] Figure 6 is a diagram showing an implementation example of an SNN circuit according to a first embodiment of the present technology.

[0029] Figure 7 is a block diagram showing a configuration example of a core according to a first embodiment of the present technology.

[0030] Figure 8 is a diagram for explaining a method of generating a readout control signal according to a first embodiment of the present technology.

[0031] Figure 9 is a block diagram showing a configuration example of a test pattern generation unit according to a first embodiment of the present technology.

[0032] Figure 10 is a flowchart showing an example of the operation of an optical detection device according to a first embodiment of the present technology.

[0033] Figure 11 is a block diagram showing a configuration example of an SNN processor according to a first modification of a first embodiment of the present technology.

[0034] Figure 12 is a diagram showing an example of a status line in a second modification of a first embodiment of the present technology.

[0035] Figure 13 is a diagram showing an example of a stacked structure of sensor chips according to a third modification of a first embodiment of the present technology.

[0036] Figure 14 is a circuit diagram showing an example of the configuration of pixels in a third modification of a first embodiment of the present technology.

[0037] Figure 15 is a diagram showing an example of a stacked structure of sensor chips according to a fourth modification of a first embodiment of the present technology.

[0038] Figure 16 is a block diagram showing a configuration example of a sensor chip according to a second embodiment of the present technology.

[0039] Figure 17 is a block diagram showing a configuration example of an SNN processor according to a second embodiment of the present technology.

[0040] Figure 18 is a block diagram showing a configuration example of a sensor chip according to a modification of a second embodiment of the present technology.

[0041] Figure 19 is a block diagram showing a configuration example of a sensor chip according to a third embodiment of the present technology.

[0042] Figure 20 is a block diagram showing a configuration example of a photon measurement circuit according to a third embodiment of the present technology.

[0043] Figure 21 is a circuit diagram showing an example of the configuration of pixels according to a third embodiment of the present technology.

[0044] Figure 22It is a block diagram showing a configuration example of a sensor chip according to a fourth embodiment of the present technology.

[0045] Figure 23 It is a block diagram showing a configuration example of a CIS according to a fourth embodiment of the present technology.

[0046] Figure 24 It is a circuit diagram showing a configuration example of a pixel according to a fourth embodiment of the present technology.

[0047] Figure 25 It is a block diagram showing a schematic configuration example of a vehicle control system.

[0048] Figure 26 It is an explanatory diagram showing an example of the installation position of an imaging unit. Detailed Embodiments

[0049] Hereinafter, modes for implementing the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order.

[0050] 1. First Embodiment (Example of Performing Readout Control Based on the Output of an SNN Circuit)

[0051] 2. Second Embodiment (Example of the Digital Processing Unit Performing Readout Control Based on the Output of an SNN Circuit)

[0052] 3. Third Embodiment (Example of Performing Readout Control of a Photon Counting Circuit Based on the Output of an SNN Circuit)

[0053] 4. Fourth Embodiment (Example of Performing CIS Readout Control Based on the Output of an SNN Circuit)

[0054] 5. Application Example of a Moving Body

[0055] <1. First Embodiment>

[0056] [Configuration Example of a Light Detection Device]

[0057] Figure 1 It is a block diagram showing a configuration example of a light detection device 100 according to a first embodiment of the present technology. The light detection device 100 includes an optical unit 110, a sensor chip 200, and a digital signal processing (DSP) circuit 120. The light detection device 100 further includes a display unit 130, an operation unit 140, a bus 150, a frame memory 160, a storage unit 170, and a power supply unit 180. As the light detection device 100, in addition to digital cameras such as digital still cameras, for example, smartphones, personal computers, in-vehicle cameras, etc. are also assumed.

[0058] The optical unit 110 gathers light from an object and guides the light to the sensor chip 200. The sensor chip 200 generates and processes a plurality of pixel data through photoelectric conversion. The sensor chip 200 provides the processed data to the DSP circuit 120.

[0059] The DSP circuit 120 performs predetermined signal processing on the data from the sensor chip 200. The DSP circuit 120 outputs the processed data to the frame memory 160 etc. via the bus 150.

[0060] The display unit 130 displays image data etc. As the display unit 130, for example, a liquid crystal panel or an organic electroluminescence (EL) panel is assumed. The operation unit 140 generates an operation signal according to a user's operation.

[0061] The bus 150 is a common path through which the optical unit 110, the sensor chip 200, the DSP circuit 120, the display unit 130, the operation unit 140, the frame memory 160, the storage unit 170, and the power supply unit 180 exchange data with each other.

[0062] The storage unit 170 stores various types of data, such as image data. The power supply unit 180 supplies power to the sensor chip 200, the DSP circuit 120, the display unit 130, etc.

[0063] [Configuration example of sensor chip]

[0064] Figure 2 is a block diagram showing a configuration example of the sensor chip 200 according to the first embodiment of the present technology. The sensor chip 200 is a single semiconductor chip, and includes an EVS 300 and an SNN processor 500. In addition, the sensor chip 200 includes FIFO memories 211 and 212, a test mode generation unit 220, a digital processing unit 241, a format processing unit 251, and an external communication interface 261.

[0065] The EVS 300 detects the brightness change of each pixel. The EVS 300 sequentially selects a plurality of lines in a pixel array unit (not shown), and the pixel data of each pixel in the line is arranged as data of a pixel line (PL). Then, the EVS 300 outputs each PL to the FIFO memory 211. Each of the pixel data includes, for example, bits representing the detection result of the brightness change of the pixel. Note that the EVS 300 is an example of the sensor recited in the claims. In addition, the PL is an example of the sensor data recited in the claims.

[0066] The FIFO memory 211 holds the PL from the EVS 300 in a first-in-first-out manner. The PL is read by the test mode generation unit 220 and the SNN processor 500. Note that the FIFO memory 211 is an example of the first FIFO memory described in the claims.

[0067] The SNN processor 500 processes the PL based on the SNN model and generates a read control signal Ctrl based on the processing result. The read control signal Ctrl is a control signal indicating a read target pixel group in the pixel array unit of the EVS 300. The SNN processor 500 outputs the read control signal Ctrl to the FIFO memory 212.

[0068] The FIFO memory 212 holds the read control signal Ctrl from the SNN processor 500 in a first-in-first-out manner. The read control signal Ctrl is read by the EVS 300. Note that the FIFO memory 212 is an example of the second FIFO memory described in the claims.

[0069] The test mode generation unit 220 generates a predetermined test mode in the test mode. The test mode generation unit 220 provides the test mode to the digital processing unit 241 in the case of the test mode and provides the PL to the digital processing unit 241 in the case of the non-test mode.

[0070] Note that the test mode generation unit 220 is set as needed. In the case where the test mode generation unit 220 is not required, the PL from the FIFO memory 211 is directly input to the digital processing unit 241.

[0071] The digital processing unit 241 performs various types of digital processing on the PL. The digital processing unit 241 provides the processed PL to the format processing unit 251.

[0072] The format processing unit 251 generates a communication frame for storing the PL. The format processing unit 251 provides the generated communication frame to the external communication interface 261.

[0073] The external communication interface 261 transmits the communication frame from the format processing unit 251 to the DSP circuit 120 etc. For example, the Mobile Industry Processor Interface (MIPI) is used as the communication standard of the external communication interface 261.

[0074] [Configuration example of EVS]

[0075] Figure 3 is a block diagram showing a configuration example of the EVS 300 according to the first embodiment of the present technology. The EVS 300 includes a driving unit 310, a pixel array unit 320, a timing control circuit 330, and a line scanner 340. In the pixel array unit 320, a plurality of pixels 400 are arranged in a two-dimensional lattice pattern.

[0076] The driving unit 310 drives each of the pixels 400. The pixels 400 detect the presence or absence of a change in luminance and generate pixel data representing the detection result.

[0077] The timing control circuit 330 controls the timing for controlling the driving unit 310 and the line scanner 340. A vertical synchronization signal is input to the timing control circuit 330. The timing control circuit 330 generates a horizontal synchronization signal based on the vertical synchronization signal and supplies the horizontal synchronization signal to the line scanner 340.

[0078] The line scanner 340 sequentially selects lines (rows, columns, etc.) in synchronization with the horizontal synchronization signal and reads the pixel data of each pixel in the line. The line scanner 340 arranges the pixel data read from the line in one dimension and outputs the data as PL to the FIFO memory 211. Note that the readout unit is a line unit, but it can also be a region unit. In this case, the line scanner 340 arranges the pixel data of each pixel read from the selected region in one dimension in a predetermined order and outputs the pixel data as PL.

[0079] In addition, the driving unit 310 and the line scanner 340 select the rows and columns to be read according to the readout control signal Ctrl from the FIFO memory 212. The readout control signal Ctrl indicates a pixel group to be read, for example, in units of rows or columns. Note that the readout control signal Ctrl can also indicate a pixel group to be read in units of regions. In the initial state, all pixels are read.

[0080] The control for sequentially reading pixel data in units of lines and regions in synchronization with a synchronization signal (such as a horizontal synchronization signal) as shown in this figure is called a scanning method. Note that, as described later, the EVS 300 can also use an arbiter method for reading pixel data without synchronization with a synchronization signal.

[0081] [Configuration example of pixel]

[0082] Figure 4 is a circuit diagram showing a configuration example of the pixel 400 according to the first embodiment of the present technology. The pixel 400 includes a pixel circuit 410, a buffer 420, a differentiating circuit 430, and a quantizer 440.

[0083] The pixel circuit 410 includes a photodiode 411, negative-channel MOS (nMOS) transistors 412 and 413, and a positive-channel MOS (pMOS) transistor 414.

[0084] The photodiode 411 generates a photocurrent by photoelectrically converting incident light. The nMOS transistor 412 is inserted between the power supply and the photodiode 411. The pMOS transistor 414 and the nMOS transistor 413 are connected in series between the power supply and the ground terminal. In addition, the gate of the nMOS transistor 413 is connected to the connection point of the nMOS transistor 412 and the photodiode 411, and the bias voltage Vblog is applied to the gate of the pMOS transistor 414.

[0085] The buffer 420 includes pMOS transistors 421 and 422 connected in series between the power supply and the ground terminal. The gate of the ground-side pMOS transistor 422 is connected to the connection point of the pMOS transistor 414 and the nMOS transistor 413. The bias voltage Vbsf is applied to the gate of the power-supply-side pMOS transistor 421. In addition, the connection point of the pMOS transistors 421 and 422 is connected to the differential circuit 430.

[0086] A voltage signal according to the photocurrent is generated and output from the buffer 420 through the above circuit.

[0087] The differential circuit 430 includes capacitors 431 and 433, pMOS transistors 432 and 434, and an nMOS transistor 435.

[0088] One end of the capacitor 431 is connected to the buffer 420, and the other end is connected to one end of the capacitor 433 and the gate of the pMOS transistor 434. The reset signal xrst is input to the gate of the pMOS transistor 432, and the source and drain are connected to both ends of the capacitor 433. The pMOS transistor 434 and the nMOS transistor 435 are connected in series between the power supply and the ground terminal. In addition, the other end of the capacitor 433 is connected to the connection point of the pMOS transistor 434 and the nMOS transistor 435. The bias voltage Vba is applied to the gate of the ground-side nMOS transistor 435, and the connection point of the pMOS transistor 434 and the nMOS transistor 435 is also connected to the quantizer 440. Through this connection, a differential signal representing the change amount of the voltage signal is generated and output to the quantizer 440. In addition, the differential signal is initialized by the reset signal xrst.

[0089] The quantizer 440 includes a pMOS transistor 441 and an nMOS transistor 442 connected in series between the power supply and the ground terminal. The gate of the pMOS transistor 441 is connected to the differential circuit 430, and a predetermined upper threshold Vbon is applied to the gate of the nMOS transistor 442. The voltage signal at the connection point between the pMOS transistor 441 and the nMOS transistor 442 is read by the line scanner 340 as a detection signal of the luminance change.

[0090] In this figure, an ON event is detected when the differential signal representing the luminance change exceeds the upper threshold Vbon. Note that the pixel 400 can also detect an OFF event when the differential signal drops below the lower threshold Vboff. In this case, a pMOS transistor 443 and an nMOS transistor 444 connected in series between the power supply and the ground terminal are added. The gate of the pMOS transistor 443 is connected to the differential circuit 430, and the lower threshold Vboff is applied to the gate of the nMOS transistor 444. The pixel 400 can detect both ON events and OFF events, or can detect only one of them.

[0091] [Configuration example of SNN processor]

[0092] Figure 5 FIG. is a block diagram showing a configuration example of an SNN processor 500 according to a first embodiment of the present technology. The SNN processor 500 includes an SNN circuit 510 and a readout control unit 550.

[0093] As described above, the EVS 300 reads the PL from the pixel array unit. The PL is input to the SNN circuit 510 via the FIFO memory 211. The SNN circuit 510 processes the PL based on the SNN model and generates line data in which a plurality of processing results are arranged as a pulse line (SL). The SL is data that arranges the pulse signals output from a row of neurons in the SNN circuit 510 at a certain moment. The SNN circuit 510 outputs the SL to the readout control unit 550.

[0094] The readout control unit 550 generates a readout control signal Ctrl based on the SL. According to the readout control signal Ctrl, the EVS 300 reads the next PL.

[0095] With the configuration shown in this figure, for example, an application in which the SNN processor 500 recognizes a predetermined object in an image and designates a region of interest (ROI) including the object by the readout control signal Ctrl can be realized.

[0096] When the output rate of the PL is high, the bandwidth between the EVS 300 and the FIFO memory 211 is insufficient, and processing delays may occur in the test mode generation unit 220 and subsequent parts. In addition, there is a possibility that the power consumption of the circuit after the test mode generation unit 220 increases.

[0097] However, since the SNN processor 500 designates the next pixel group to be read by the readout control signal Ctrl, the output rate can be reduced compared to the case of reading all pixels, and an increase in processing delay and power consumption can be suppressed. Therefore, the performance of the optical detection device 100 can be improved.

[0098] In addition, the SNN circuit 510 includes an input layer 520, an intermediate layer 530, and an output layer 540.

[0099] The PL is input to the input layer 520. One or more layers are provided in the intermediate layer 530. Neurons in the previous layer are connected to neurons in the next layer, and the operation results of the previous layer are passed to the next layer. The output layer 540 asynchronously generates pulse signals.

[0100] In the output layer 540, for example, a pair of neuron rows are provided. One neuron row outputs the SL as SLa at each time point, and the other neuron row outputs the SL as SLb at each time point. The read control unit 550 compares the output data of each neuron row and generates a read control signal Ctrl based on the comparison result. Note that SLa is an example of the first line data described in the claims, and SLb is an example of the second line data described in the claims.

[0101] Figure 6 is a diagram showing an implementation example of the SNN circuit 510 according to the first embodiment of the present technology. Figure 5 The SNN circuit 510 of Figure 6 is implemented by a circuit such as Figure 6 As shown, the SNN circuit 510 includes, for example, an input / output interface 560 and a multi-core array 570.

[0102] The input / output interface 560 transmits and receives data between the outside and the multi-core array 570. The input / output interface 560 provides the PL input from the FIFO memory 211 to the multi-core array 570 and provides the SL from the multi-core array 570 to the read control unit 550.

[0103] In the multi-core array 570, a plurality of cores 590 are arranged in a two-dimensional lattice pattern. The router 580 is provided adjacent to each of the cores 590.

[0104] The router 580 controls the path of data. The router 580 includes, for example, FIFO memories 581 to 585 and an arbiter 586. In this figure, "E" represents the east direction of the router 580 of interest, and "S" represents the south direction. "W" represents the west direction, and "N" represents the north direction. "L" represents the direction toward the core 590 adjacent to the router 580.

[0105] The FIFO memory 581 holds data from the east direction in a first-in-first-out manner and outputs a request to the arbiter 586. The FIFO memory 582 holds data from the south direction in a first-in-first-out manner and outputs a request to the arbiter 586. The FIFO memory 583 holds data from the west direction in a first-in-first-out manner and outputs a request to the arbiter 586. The FIFO memory 584 holds data from the north direction in a first-in-first-out manner and outputs a request to the arbiter 586. The FIFO memory 585 holds data from the adjacent core 590 in a first-in-first-out manner and outputs a request to the arbiter 586.

[0106] Note that in the SNN circuit 510, the external FIFO memory 211 can be reduced and replaced with the FIFO memory 581 or the like.

[0107] The arbiter 586 arbitrates requests from each of the FIFO memories 581 to 585 and returns a response. When there is a response, the FIFO memory outputs data to one of the cores 590 adjacent to the north, south, east, and west via the arbiter 586.

[0108] Figure 7 is a block diagram showing a configuration example of the core 590 according to the first embodiment of the present technology. The core 590 includes a core router 591, a neuron input / output (I / O) 592, a multiply-accumulate unit 593, a working memory 594, a membrane potential memory 595, and a leaky integrate-and-fire (LIF) unit 596.

[0109] The core router 591 provides data from the adjacent router 580 to the neuron I / O 592 and provides data from the LIF unit 596 to the adjacent router 580.

[0110] The multiply-accumulate unit 593 integrates data from the neuron I / O 592 using the working memory 594. The membrane potential memory 595 holds the membrane potential obtained by integration. The LIF unit 596 determines whether the membrane potential exceeds a predetermined threshold and is fired (in other words, a pulse appears) and provides the result to the core router 591.

[0111] Figure 8 is a diagram for explaining a method of generating a read control signal according to the first embodiment of the present technology.

[0112] As shown in a of this figure, a plurality of PLs are sequentially input to the input layer 520. Each PL includes a plurality of pixel data. For example, each pixel data is 1-bit information indicating whether an on event has been detected. In this figure, x0 to xj represent the x coordinates of each pixel in the line.

[0113] In this figure, b shows a configuration example of the output layer 540. In the output layer 540, a pair of neurons is set for each line. When the number of lines is k, a neuron row Ra arranged with neurons 541-1 to 541-k and a neuron row Rb arranged with neurons 542-1 to 542-k are set. The data output from neurons 541-1 to 541-k are 1a to ka, and the data output from neurons 542-1 to 542-k are 1b to kb.

[0114] The data output from each neuron includes, for example, pulse groups C1a to Cja generated in mutually different time periods. The pulse groups C1a to Cja are data corresponding to the respective pixels of the x coordinates x1 to xj. Each of the pulse groups also includes a plurality of pulse signals generated within the corresponding time period. In this figure, t1 to tm represent the moments when the pulse signals are generated. In addition, white rectangles indicate the presence of pulses, while black rectangles indicate the absence of pulses.

[0115] At a certain moment, the data arranging the pulse signals output from each neuron in the neuron row Ra corresponds to the above-mentioned SLa. At a certain moment, the data arranging the pulse signals output from each neuron in the neuron row Rb corresponds to the above-mentioned SLb.

[0116] The read control unit 550 compares the data output from each of the neuron pairs corresponding to the lines and generates a read control signal Ctrl. The read control unit 550 counts the number of pulses for each pulse group of the neuron pairs and compares them. For example, when the pattern to be recognized occurs in a certain line, the count value of the pulse group of one of the neuron pairs (such as 541-1) corresponding to that line is set to be greater than the count value of the pulse group of the other (such as 542-1). In this case, for example, when the count value of C1a in a certain line is greater than the count value of C1b, the pixel of the x coordinate x1 of that line is designated as the read target.

[0117] Note that although the read control unit 550 compares the count values of the pulse groups, it is also possible to input the pulse groups to be compared into the softmax function and compare the output values. In addition, each of the pulse groups has a one-to-one correspondence with the pixel, but it is also possible to have a one-to-many correspondence. Furthermore, the read control unit 550 can also adjust the frequencies of reading rows and columns based on the comparison results.

[0118] The read control unit 550 generates and outputs a read control signal Ctrl for each line of the y coordinates y1 to yk. Therefore, the read control unit 550 can indicate the read target pixels in units of rows and columns.

[0119] [Configuration example of the test pattern generation unit]

[0120] Figure 9 It is a block diagram showing a configuration example of a test pattern generation unit 220 according to a first embodiment of the present technology. The test pattern generation unit 220 includes a test pattern providing unit 221 and a switch 222.

[0121] When the test state is set by the control signal "STATE (MODE)", the test pattern providing unit 221 generates a predetermined test pattern and provides the predetermined test pattern to the switch 222.

[0122] The switch 222 provides the test pattern to the digital processing unit 241 when the test state is set, and provides the PL from the FIFO memory 211 to the digital processing unit 241 when a state other than the test state is set.

[0123] [Operation example of the optical detection device]

[0124] Figure 10 It is a flowchart showing an operation example of the optical detection device 100 according to a first embodiment of the present technology. For example, when a predetermined application for capturing image data is executed, the operation starts.

[0125] The EVS 300 sequentially reads the PL according to the read control signal (step S901). Note that in the initial state, all pixels are read. In addition, the SNN processor 500 generates the SL (step S902) and generates the read control signal (step S903). In addition, the digital processing unit 241 performs digital processing on each of the PLs (step S904). In addition, the format processing unit 251 generates a communication frame by format processing (step S905), and the external communication interface 261 transmits the communication frame to the outside (step S906). After step S906, steps S901 and subsequent steps are repeatedly executed.

[0126] As described above, according to the first embodiment of the present technology, since the read control unit 550 generates the read control signal based on the SL, the output rate can be reduced and an increase in processing delay and power consumption can be suppressed. Therefore, the performance of the optical detection device 100 can be improved.

[0127] [First modification example]

[0128] In the above first embodiment, only the PL is input to the SNN processor 500, but in order to enable the SNN processor 500 to recognize the line corresponding to the PL, identification information of the line (such as a line number) may also be input. The optical detection device 100 of the first modification example of the first embodiment is different from the optical detection device of the first embodiment in that the EVS 300 inputs the line identification information and the PL to the SNN processor 500.

[0129] Figure 11 It is a block diagram showing a configuration example of the SNN processor 500 in a first modification of the first embodiment of the present technology.

[0130] In the first modification of the first embodiment, the EVS 300 reads the PL by an arbiter method or a scanning method. Then, the EVS 300 adds identification information (such as a line number) of the corresponding line for each PL, and inputs the information into the SNN processor 500 via the FIFO memory 211.

[0131] In addition, the SNN processor 500 further includes a conversion unit 505. The conversion unit 505 converts at least one of the identification information Id and the corresponding PL, and provides the identification information Id and the corresponding PL to the SNN circuit 510.

[0132] For example, the conversion unit 505 inputs the PL as it is into the SNN circuit 510, and converts the identification information Id into vector data or the like for input.

[0133] Alternatively, the conversion unit 505 inputs the identification information Id as it is into the SNN circuit 510, and converts the PL into frequency information (phase value, scalar value, etc.) by Fourier transform or the like for input.

[0134] Note that the conversion unit 505 can also convert both the identification information Id and the PL.

[0135] As shown in this figure, the conversion unit 505 converts at least one of the identification information and the PL, and inputs the converted information into the SNN circuit 510, so that the subsequent read control unit 550 can specify the line corresponding to the SL according to the identification information. Therefore, the read control unit 550 can generate a read control signal.

[0136] As described above, according to the first modification of the first embodiment of the present technology, since the conversion unit 505 converts at least one of the identification information and the PL, and inputs the converted information into the SNN circuit 510, the read control unit 550 can generate a read control signal based on the data.

[0137] [Second Modification]

[0138] In the above first embodiment, the SNN circuit 510 outputs the SL, but it can also output the state value of the membrane potential in chronological order instead of the SL. The optical detection device 100 in the second modification of the first embodiment is different from the optical detection device in the first embodiment in that the SNN circuit 510 outputs the state value of the membrane potential in chronological order.

[0139] Figure 12FIG. is a diagram showing an example of a state line in a second modification of the first embodiment of the present technology. In the second modification of the first embodiment, pairs such as neurons 541-1 and 542-1 output output data Da and Db. Each of the output data includes a plurality of bit strings representing state values of membrane potentials in chronological order. In this figure, t1 to tj represent the moments when the bit strings are output. The rectangles in the lower part of the moments are bit strings of two or more bits representing the state values at that moment. In addition, the darker the color of the rectangle, the larger the state value.

[0140] In addition, the state values from moment t1 to moment tj are values corresponding to the pixels of the x coordinates x1 to xj. The read control unit 550 compares the state values of the same moment of each neuron and determines whether to read the pixel corresponding to that moment based on the comparison result.

[0141] Note that the first modification can be applied to the second modification of the first embodiment.

[0142] As described above, according to the second modification of the first embodiment of the present technology, since the SNN circuit 510 outputs the state values of the membrane potential in chronological order, the read control unit 550 can generate a read control signal based on these state values.

[0143] [Third Modification]

[0144] In the above first embodiment, a circuit such as the EVS 300 is provided on a single semiconductor chip, but in this configuration, it may be difficult to increase the number of pixels. The optical detection device 100 according to the third modification of the first embodiment is different from the optical detection device of the first embodiment in that the circuits are dispersedly provided on two stacked semiconductor chips.

[0145] Figure 13 FIG. is a diagram showing an example of the stacked structure of the sensor chip 200 in the third modification of the first embodiment of the present technology. The sensor chip 200 of the third modification of the first embodiment includes a pixel chip 201 and a circuit chip 202. These chips are stacked and electrically connected by, for example, Cu-Cu bonding. Note that in addition to Cu-Cu bonding, vias or bumps can also be used for connection.

[0146] Figure 14 FIG. is a circuit diagram showing an example of the configuration of the pixel 400 in the third modification of the first embodiment of the present technology. In the pixel 400, for example, the pixel circuit 410 is provided on the pixel chip 201, and the circuit after the buffer 420 is provided on the circuit chip 202.

[0147] Note that the circuits arranged on each chip are not limited to those shown in this figure. For example, the photodiode 411 and the nMOS transistors 412 and 413 can be provided in the pixel chip 201, and the remaining circuits can be provided in the circuit chip 202. Alternatively, only the photodiode 411 can be provided in the pixel chip 201, and the remaining circuits can be provided in the circuit chip 202.

[0148] Note that each of the first modification example and the second modification example can be applied to the third modification example of the first embodiment.

[0149] As described above, according to the third modification example of the first embodiment of the present technology, since the circuits are dispersedly arranged in two stacked chips, the circuit scale of each chip can be reduced. This helps to increase the number of pixels.

[0150] [Fourth Modification Example]

[0151] In the above first embodiment, circuits such as the EVS 300 are provided on a single semiconductor chip. However, in this configuration, it may be difficult to increase the number of pixels. The light detection device 100 according to the fourth modification example of the first embodiment is different from the light detection device of the first embodiment in that the circuits are dispersedly arranged on three stacked semiconductor chips.

[0152] Figure 15 FIG. is a diagram showing an example of the stacked structure of the sensor chip 200 in the fourth modification example of the first embodiment of the present technology. In the fourth modification example of the first embodiment, the sensor chip 200 includes a stacked pixel chip 201, a circuit chip 202, and a circuit chip 203. A part of the pixels of the EVS 300 (such as the pixel circuit 410) is provided on the pixel chip 201, and the remaining circuits of the EVS 300 are provided on the circuit chip 202. In addition, the circuits after the FIFO memory 211 are provided in the circuit chip 203. Note that the circuits arranged on each chip are not limited to those shown in this figure. In addition, the number of stacked chips is not limited to three and can be four or more.

[0153] Note that each of the first modification example and the second modification example can be applied to the fourth modification example of the first embodiment.

[0154] As described above, according to the fourth modification example of the first embodiment of the present technology, since the circuits are dispersedly arranged in three stacked chips, the circuit scale of each chip can be reduced. This helps to increase the number of pixels.

[0155] <2. Second Embodiment>

[0156] In the above first embodiment, the SNN processor 500 generates a read control signal. However, in this configuration, it is necessary to add a read control unit 550 to the SNN processor 500. The optical detection device 100 in the second embodiment is different from the optical detection device in the first embodiment in that the subsequent digital processing unit generates the read control signal.

[0157] Figure 16 FIG. is a block diagram showing a configuration example of a sensor chip 200 according to a second embodiment of the present technology. The sensor chip 200 in the second embodiment is different from the sensor chip in the first embodiment in that it further includes a test mode generation unit 230 and a digital processing unit 242.

[0158] In the second embodiment, the SNN processor 500 outputs the SL to the FIFO memory 212 without generating a read control signal Ctrl.

[0159] The test mode generation unit 220 provides a test mode to the digital processing unit 241 in the case of a test state, and reads the SL from the FIFO memory 212 and provides the SL to the digital processing unit 242 in the case of a non-test state.

[0160] The digital processing unit 242 generates a read control signal Ctrl based on the SL and outputs the read control signal Ctrl to the EVS 300. Note that the digital processing unit 242 is an example of the read control unit recited in the claims.

[0161] Figure 17 FIG. is a block diagram showing a configuration example of an SNN processor 500 according to a second embodiment of the present technology. As shown in this figure, the SNN processor 500 in the second embodiment is different from the SNN processor in the first embodiment in that the read control unit 550 is not provided.

[0162] As Figure 16 and Figure 17 shown, the digital processing unit 242 generates a read control signal, enabling the reduction of the read control unit 550 in the SNN processor 500.

[0163] Note that each of the first to fourth modification examples of the first embodiment can be applied to the second embodiment.

[0164] As described above, according to the second embodiment of the present technology, since the digital processing unit 242 generates the read control signal instead of the SNN processor 500, the circuit scale of the SNN processor 500 can be reduced.

[0165] [Modification Example]

[0166] In the above second embodiment, the sensor chip 200 does not output the SL and its processing result to the outside, but it is also possible to output the SL and its processing result to the outside. The sensor chip 200 according to the modified example of the second embodiment is different from the sensor chip of the first embodiment in that it outputs the SL, etc. to the outside.

[0167] Figure 18 FIG. is a block diagram showing a configuration example of the sensor chip 200 in a modified example of the second embodiment of the present technology. The sensor chip 200 in the modified example of the second embodiment is different from the second embodiment in that it further includes a format processing unit 252 and an external communication interface 262.

[0168] In addition, in the second embodiment, the digital processing unit 242 generates a read control signal and performs various digital processes on the SL as needed. The digital processing unit 242 outputs the processed SL to the format processing unit 252.

[0169] The format processing unit 252 generates a communication frame for storing the SL, etc. The format processing unit 252 provides the generated communication frame to the external communication interface 262.

[0170] The external communication interface 262 transmits the communication frame from the format processing unit 252 to the DSP circuit 120, etc.

[0171] Note that each of the first to fourth modified examples of the first embodiment can be applied to the modified example of the second embodiment.

[0172] As described above, according to the modified example of the second embodiment of the present technology, since the sensor chip 200 further outputs the SL to the outside, the circuit outside the sensor chip 200 can use this data.

[0173] <3. Third Embodiment>

[0174] In the above first embodiment, the EVS 300 is used as the sensor for generating the PL, but a photon measurement circuit that counts photons can be used instead of the EVS 300. The light detection device 100 according to the third embodiment is different from the light detection device of the first embodiment in that a photon measurement circuit is used instead of the EVS 300.

[0175] Figure 19 FIG. is a block diagram showing a configuration example of the sensor chip 200 according to the third embodiment of the present technology. The sensor chip 200 in the third embodiment is different from the sensor chip in the first embodiment in that a photon measurement circuit 600 is provided instead of the EVS 300. Note that the photon measurement circuit 600 is an example of the sensor recited in the claims.

[0176] Figure 20 FIG. 6 is a block diagram showing a configuration example of a photon measurement circuit 600 according to a third embodiment of the present technology. The photon measurement circuit 600 includes a driving unit 610, a pixel array unit 620, a timing control circuit 640, and a readout processing unit 650. In the pixel array unit 620, a plurality of pixels 630 are arranged in a two-dimensional lattice pattern.

[0177] The functions of the driving unit 610, the pixel array unit 620, the timing control circuit 640, and the readout processing unit 650 are similar to the functions of the driving unit 310, the pixel array unit 320, the timing control circuit 330, and the line scanner 340.

[0178] Figure 21 FIG. 7 is a circuit diagram showing a configuration example of a pixel 630 according to a third embodiment of the present technology. The pixel 630 includes a quenching resistor 631, a single photon avalanche diode (SPAD) 632, an inverter 633, and a photon counter 634.

[0179] The quenching resistor 631 and the SPAD 632 are connected in series. The inverter 633 inverts the voltage signal at the connection point between the quenching resistor 631 and the SPAD 632, and provides the inverted voltage signal as a pulse signal to the photon counter 634. The photon counter 634 counts the number of pulses of the pulse signal, and provides pixel data representing the count value to the readout processing unit 650.

[0180] When counting photons, each piece of pixel data in the PL is a bit string of two or more bits representing the count value. However, as in the first embodiment, it is preferable to convert each piece of pixel data into 1-bit information. When performing the conversion, for example, a conversion circuit that converts the bit string into 1 bit for each pixel is inserted in the front stage of the SNN circuit 510.

[0181] Note that the circuit configuration of the pixel 630 is not limited to the circuit configuration shown in this figure, as long as it can count photons.

[0182] In addition, each of the first modification, the second modification, the third modification, and the fourth modification of the first embodiment, the second embodiment, and the modification of the second embodiment can be applied to the third embodiment.

[0183] As described above, according to the third embodiment of the present technology, since the photon measurement circuit 600 is provided instead of the EVS 300, an increase in processing delay and power consumption can be suppressed at the subsequent stage of the photon measurement circuit 600.

[0184] <4. Fourth Embodiment>

[0185] In the first embodiment described above, the EVS 300 is used as the sensor for generating the PL, but the CIS can also be used instead of the EVS 300. The optical detection device 100 according to the fourth embodiment is different from the optical detection device of the first embodiment in that the CIS is used instead of the EVS 300.

[0186] Figure 22 FIG. is a block diagram showing a configuration example of the sensor chip 200 according to the fourth embodiment of the present technology. The sensor chip 200 in the fourth embodiment is different from the sensor chip in the first embodiment in that the CIS 700 is provided instead of the EVS 300. Note that the CIS 700 is an example of the sensor described in the claims.

[0187] Figure 23 FIG. is a block diagram showing a configuration example of the CIS 700 according to the first embodiment of the present technology. The CIS 700 includes a vertical scanning circuit 710, a timing control circuit 720, a digital-to-analog converter (DAC) 730, a pixel array section 740, a column ADC 760, and a horizontal transfer scanning circuit 770. In the pixel array section 740, the pixels 750 are arranged in a two-dimensional lattice pattern.

[0188] The vertical scanning circuit 710 sequentially selects and drives the rows, and outputs an analog pixel signal to the column ADC 760. The timing control circuit 720 generates a horizontal synchronization signal based on the vertical synchronization signal, and supplies the horizontal synchronization signal to the horizontal transfer scanning circuit 770.

[0189] The DAC 730 generates a predetermined reference signal and supplies the generated reference signal to the column ADC 760. For example, a sawtooth ramp signal is used as the reference signal.

[0190] The column ADC 760 includes an ADC for each column, and performs analog-to-digital (AD) conversion on each pixel signal of the column. The column ADC 760 generates the PL based on the control of the horizontal transfer scanning circuit 770 and outputs the PL to the FIFO memory 211.

[0191] The horizontal transfer scanning circuit 770 controls the column ADC 760 to sequentially output pixel data.

[0192] As described above, in the CIS, each piece of pixel data in the PL is a bit string of two or more bits representing the gray value of the pixel. However, as in the first embodiment, it is preferable to convert each piece of pixel data into 1-bit information. In the case of performing the conversion, for example, a conversion circuit that converts the bit string into 1 bit for each pixel is inserted in the front stage of the SNN circuit 510.

[0193] Figure 24This is a circuit diagram showing a configuration example of pixel 750 according to the fourth embodiment of the present technology. Pixel 750 includes a photodiode 751, a transfer transistor 752, a reset transistor 753, a floating diffusion layer 754, an amplification transistor 755, and a selection transistor 756.

[0194] The photodiode 751 performs photoelectric conversion on incident light to generate charges. The transfer transistor 752 transfers the charges from the photodiode 751 to the floating diffusion layer 754 according to the transfer signal TRG from the vertical scanning circuit 710.

[0195] The reset transistor 753 extracts charges from the floating diffusion layer 754 according to the reset signal RST from the vertical scanning circuit 710 and initializes the charges. The floating diffusion layer 754 accumulates charges and generates a voltage corresponding to the amount of charges.

[0196] The amplification transistor 755 amplifies the voltage of the floating diffusion layer 754. The selection transistor 756 outputs the signal of the amplified voltage as a pixel signal according to the selection signal SEL from the vertical scanning circuit 710.

[0197] In addition, in the pixel array unit 740, the vertical signal line 759 is wired for each column, and the pixel signal of each pixel 750 in the column is output to the column ADC 760 via the vertical signal line 759 of the column.

[0198] Note that the circuit configuration of pixel 750 is not limited to the configuration shown in this figure, as long as an analog pixel signal can be generated.

[0199] In addition, each of the first modification, the second modification, the third modification, and the fourth modification of the first embodiment, the second embodiment, and the modification of the second embodiment can be applied to the fourth embodiment.

[0200] As described above, according to the fourth embodiment of the present technology, since the CIS 700 is provided instead of the EVS 300, an increase in processing delay and power consumption can be suppressed at the subsequent stage of the CIS 700.

[0201] <5. Application Examples of Moving Bodies>

[0202] The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the embodiments of the present disclosure can also be implemented as a device installed on any type of moving body (such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility tool, an airplane, a drone, a ship, and a robot).

[0203] Figure 25It is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving body control system to which the technology according to the present disclosure can be applied.

[0204] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Among Figure 25 the illustrated example, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside vehicle information detection unit 12030, an inside vehicle information detection unit 12040, and an integrated control unit 12050. In addition, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, a sound / image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown.

[0205] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 serves as a control device for a driving force generation device (such as an internal combustion engine, a drive motor, etc.) for generating the driving force of the vehicle, a driving force transmission mechanism for transmitting the driving force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, etc.

[0206] The body system control unit 12020 controls the operation of various devices disposed on the vehicle body according to various programs. For example, the body system control unit 12020 serves as a control device for a keyless entry system, a smart key system, an electric window device, or various lights such as headlights, rear lights, brake lights, turn signals, fog lights, etc. In this case, radio waves transmitted from a mobile device as an alternative to a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals and controls the door lock device, electric window device, lights, etc. of the vehicle.

[0207] The outside vehicle information detection unit 12030 detects information on the outside of the vehicle including the vehicle control system 12000. For example, the outside vehicle information detection unit 12030 is connected to an imaging unit 12031. The outside vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image of the outside of the vehicle and receives the captured image. Based on the received image, the outside vehicle information detection unit 12030 can perform processing for detecting objects (such as people, vehicles, obstacles, signs, words on the road surface, etc.) or processing for detecting the distance to the objects.

[0208] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of received light. The imaging unit 12031 can output the electrical signal as an image, or can output the electrical signal as information regarding the measured distance. Further, the light received by the imaging unit 12031 can be visible light, or can be invisible light such as infrared light.

[0209] The in-vehicle information detection unit 12040 detects information regarding the interior of the vehicle. The in-vehicle information detection unit 12040 is connected to, for example, a driver state detection unit 12041 that detects the state of the driver. The driver state detection unit 12041 includes, for example, a camera that photographs the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's fatigue or the driver's concentration, or can determine whether the driver is dozing off.

[0210] The microcomputer 12051 can calculate control target values for the driving force generation device, the steering mechanism, or the braking device based on the information regarding the interior or exterior of the vehicle acquired by the out-vehicle information detection unit 12030 or the in-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control for implementing the functions of an advanced driver assistance system (ADAS), which functions include collision avoidance or shock absorption for the vehicle, following driving based on the following distance, maintaining vehicle speed driving, warning of vehicle collision, warning of vehicle departure from the lane, and the like.

[0211] In addition, the microcomputer 12051 can perform cooperative control for autonomous driving by controlling the driving force generation device, the steering mechanism, the braking device, etc., based on the information regarding the exterior or interior of the vehicle acquired by the out-vehicle information detection unit 12030 or the in-vehicle information detection unit 12040, so that the vehicle automatically travels without relying on the driver's operation or the like.

[0212] Further, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information regarding the exterior of the vehicle acquired by the out-vehicle information detection unit 12030. For example, the microcomputer 12051 can perform cooperative control for preventing glare by, for example, controlling the headlamp to change from high beam to low beam according to the position of the vehicle ahead or the oncoming vehicle detected by the out-vehicle information detection unit 12030.

[0213] The sound / image output unit 12052 transmits an output signal of at least one of sound and image to an output device that can visually or auditorily notify information to the occupants of the vehicle or the exterior of the vehicle. In Figure 25In the example, the audio speaker 12061, the display unit 12062, and the instrument panel 12063 are depicted as output devices. For example, the display unit 12062 may include at least one of an on-board display and a head-up display.

[0214] Figure 26 is a diagram showing an example of the installation position of the imaging unit 12031.

[0215] In Figure 26 the imaging units 12101, 12102, 12103, 12104, and 12105 are included as the imaging unit 12031.

[0216] For example, the imaging units 12101, 12102, 12103, 12104, and 12105 are disposed at positions such as the front nose, side mirrors, rear bumper, rear door, and the upper part of the windshield inside the vehicle 12100. The imaging unit 12101 disposed at the front nose and the imaging unit 12105 disposed at the upper part of the windshield inside the vehicle mainly acquire images in front of the vehicle 12100. The imaging units 12102 and 12103 disposed at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 disposed at the rear bumper or rear door mainly acquires images behind the vehicle 12100. The imaging unit 12105 disposed at the upper part of the windshield inside the vehicle is mainly used to detect a preceding vehicle, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0217] Note that Figure 26 shows examples of the imaging ranges of the imaging units 12101 to 12104. The imaging range 12111 represents the imaging range of the imaging unit 12101 disposed at the front nose. The imaging ranges 12112 and 12113 represent the imaging ranges of the imaging units 12102 and 12103 disposed at the side mirrors, respectively. The imaging range 12114 represents the imaging range of the imaging unit 12104 disposed at the rear bumper or rear door. For example, a bird's-eye view image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.

[0218] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0219] For example, the microcomputer 12051 can determine the distances to each three-dimensional object within the imaging ranges 12111 to 12114 and the temporal change of the distances (relative speed with respect to the vehicle 12100) based on the distance information acquired from the imaging units 12101 to 12104. And thereby, extract the nearest three-dimensional object traveling on the traveling path of the vehicle 12100 in substantially the same direction as the vehicle 12100 at a predetermined speed (e.g., equal to or greater than 0 km / h) as the preceding vehicle. In addition, the microcomputer 12051 can preset the following distance to be maintained in front of the preceding vehicle and execute automatic braking control (including following stop control), automatic acceleration control (including following start control), etc. Thereby, it is possible to execute coordinated control for autonomous driving that enables the vehicle to automatically travel without relying on the driver's operation, etc.

[0220] For example, the microcomputer 12051 can classify the three-dimensional object data related to the three-dimensional object into three-dimensional object data of a two-wheeled vehicle, a standard-sized vehicle, a large vehicle, a pedestrian, a utility pole, and other three-dimensional objects based on the distance information acquired from the imaging units 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatically avoiding obstacles. For example, the microcomputer 12051 identifies the obstacles around the vehicle 12100 as obstacles that can be visually recognized by the driver of the vehicle 12100 and obstacles that are difficult for the driver of the vehicle 12100 to visually recognize. Then, the microcomputer 12051 determines the collision risk indicating the risk of collision with each obstacle. When the collision risk is equal to or higher than the set value and thus there is a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display unit 12062 and executes forced deceleration or avoidance steering via the drive system control unit 12010. The microcomputer 12051 can thereby assist driving to avoid collisions.

[0221] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. The microcomputer 12051 can identify a pedestrian, for example, by determining whether a pedestrian exists in the captured images of the imaging units 12101 to 12104. Such identification of a pedestrian is performed, for example, by a process of extracting feature points in the captured images of the imaging units 12101 to 12104 that are infrared cameras and by a process of performing pattern matching processing on a series of feature points representing the contour of an object to determine whether it is a pedestrian. When the microcomputer 12051 determines that a pedestrian exists in the captured images of the imaging units 12101 to 12104 and thus identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 such that a square contour line for emphasis is displayed superimposed on the identified pedestrian. The sound / image output unit 12052 may also control the display unit 12062 such that an icon representing the pedestrian is displayed at a desired position.

[0222] Examples of a vehicle control system to which the technology according to the present disclosure can be applied have been described above. For example, the technology according to the present disclosure can be applied to the imaging unit 12031 in the above configuration. Specifically, Figure 1 the light detection device 100 can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, the performance of the system can be improved by suppressing an increase in processing delay and power consumption.

[0223] Note that the above embodiments show examples for embodying the present technology, and there is a correspondence between the matters in the embodiments and the matters specifying the present invention in the claims. Similarly, there is a correspondence between the matters specifying the present invention in the claims and the matters having the same name in the embodiments of the present technology. However, the present technology is not limited to the embodiments, and various modifications can be applied to the embodiments without departing from the gist of the present technology.

[0224] Note that the effects described in this specification are merely exemplary and not limited, and furthermore, other effects may be provided.

[0225] Note that the present technology may also have the following configuration.

[0226] (1) A light detection device, comprising:

[0227] a sensor that reads sensor data in which a plurality of pixel data are arranged from a pixel array unit;

[0228] a neural network circuit that processes the sensor data based on a neural network model and outputs line data in which a plurality of processing results are arranged; and

[0229] A readout control unit that generates a readout control signal based on line data, the readout control signal indicating a readout target pixel group in the pixel array unit.

[0230] (2) The optical detection device according to (1) above, wherein

[0231] The line data includes first line data and second line data,

[0232] The neural network circuit outputs the first line data and the second line data in parallel, and

[0233] The readout control unit compares the first line data with the second line data and generates a readout control signal based on the comparison result.

[0234] (3) The optical detection device according to (2) above, wherein

[0235] The neural network model includes a spiking neural network model,

[0236] Each of the first line data and the second line data includes a plurality of bit strings, and

[0237] Each of the plurality of bit strings includes a plurality of bits representing the detection result of a pulse in chronological order.

[0238] (4) The optical detection device according to (2) above, wherein

[0239] The neural network model includes a spiking neural network model, and

[0240] Each of the first line data and the second line data includes a plurality of bit strings representing the state values of the membrane potential in chronological order.

[0241] (5) The optical detection device according to any one of (1) to (4) above, further comprising:

[0242] A conversion unit that converts at least one of the identification information and the sensor data and provides the converted identification information and / or sensor data to the neural network circuit, wherein

[0243] The sensor outputs the identification information together with the sensor data.

[0244] (6) The optical detection device according to any one of (1) to (5) above, further comprising:

[0245] A first FIFO (First In First Out) memory that holds the sensor data in a first-in first-out manner; and

[0246] A second FIFO memory that holds the readout control signal in a first-in first-out manner, wherein

[0247] The neural network circuit reads sensor data from the first FIFO memory, and

[0248] the sensor reads the readout control signal from the second FIFO memory.

[0249] (7) The optical detection device according to any one of (1) to (5) above further includes:

[0250] a first FIFO memory that holds sensor data in a first-in, first-out manner; and

[0251] a second FIFO memory that holds line data in a first-in, first-out manner, wherein

[0252] the neural network circuit reads sensor data from the first FIFO memory, and

[0253] the readout control unit reads line data from the second FIFO memory.

[0254] (8) The optical detection device according to (7) above further includes:

[0255] a digital processing unit that reads sensor data from the first FIFO memory and processes the sensor data; and

[0256] a first format processing unit that generates a communication frame storing the sensor data.

[0257] (9) The optical detection device according to (8) above further includes:

[0258] a second format processing unit that generates a communication frame storing the line data, wherein

[0259] the readout control unit outputs the line data to the second format processing unit.

[0260] (10) The optical detection device according to any one of (1) to (9) above, wherein

[0261] the sensor includes an EVS (event-based vision sensor).

[0262] (11) The optical detection device according to any one of (1) to (9) above, wherein

[0263] the sensor includes a photon measurement circuit that counts photons.

[0264] (12) The optical detection device according to any one of (1) to (9) above, wherein

[0265] the sensor includes a CIS (CMOS image sensor).

[0266] (13) The optical detection device according to any one of (1) to (12) above, wherein

[0267] The sensor, the neural network circuit, and the readout control unit are dispersedly arranged on a plurality of stacked chips.

[0268] (14) A control method for an optical detection device, the control method comprising:

[0269] A process of reading sensor data in which a plurality of pixel data are arranged from a pixel array unit through a sensor;

[0270] A process of processing the sensor data based on a neural network model and outputting line data in which a plurality of processing results are arranged; and

[0271] A readout control process of generating a readout control signal based on the line data, the readout control signal indicating a readout target pixel group in the pixel array unit.

[0272] List of reference numerals

[0273] 100 Optical detection device

[0274] 110 Optical unit

[0275] 120 DSP circuit

[0276] 130 Display unit

[0277] 140 Operation unit

[0278] 150 Bus

[0279] 160 Frame memory

[0280] 170 Storage unit

[0281] 180 Power supply unit

[0282] 200 Sensor chip

[0283] 201 Pixel chip

[0284] 202, 203 Circuit chips

[0285] 211, 212, 581 to 585 FIFO memories

[0286] 220, 230 Test mode generation units

[0287] 221 Test mode providing unit

[0288] 222 Switch

[0289] 241, 242 Digital processing units

[0290] 251, 252 Format Processing Unit

[0291] 261, 262 External Communication Interface 300EVS

[0292] 310, 610 Driver Unit

[0293] 320, 620, 740 Pixel Array Unit

[0294] 330, 640, 720 Timing Control Circuit

[0295] 340 Line Scanner

[0296] 400, 630, 750 Pixel

[0297] 410 Pixel Circuit

[0298] 411, 751 Photoelectric Diode

[0299] 412, 413, 435, 442, 444 nMOS Transistor

[0300] 414, 421, 422, 432, 434, 441, 443 pMOS Transistor 420 Buffer

[0301] 430 Differentiating Circuit

[0302] 431, 433 Capacitor

[0303] 440 Quantizer

[0304] 500 SNN Processor

[0305] 505 Conversion Unit

[0306] 510 SNN Circuit

[0307] 520 Input Layer

[0308] 530 Intermediate Layer

[0309] 540 Output Layer

[0310] 541-1 to 541-k, 542-1 to 542-k Neuron

[0311] 550 Readout Control Unit

[0312] 560 Input / Output Interface

[0313] 570 Multi-Core Array

[0314] 580 Router

[0315] 586 Arbiter

[0316] 590 Core

[0317] 591 Core Router

[0318] 592 Neuron I / O

[0319] 593 Product-Sum Unit

[0320] 594 Working Memory

[0321] 595 Membrane Potential Memory

[0322] 596 LIF Unit

[0323] 600 Photon Measurement Circuit

[0324] 631 Quenching Resistor

[0325] 632 SPAD

[0326] 633 Inverter

[0327] 634 Photon Counter

[0328] 650 Readout Processing Unit

[0329] 700 CIS

[0330] 710 Vertical Scanning Circuit

[0331] 730 DAC

[0332] 752 Transfer Transistor

[0333] 753 Reset Transistor

[0334] 754 Floating Diffusion Layer

[0335] 755 Amplification Transistor

[0336] 756 Selection Transistor

[0337] 760 Column ADC

[0338] 770 Horizontal Transfer Scanning Circuit

[0339] 12031 Imaging Unit.

Claims

1. An optical detection device, comprising: a sensor that reads sensor data in which a plurality of pixel data are arranged from a pixel array unit; a neural network circuit that processes the sensor data based on a neural network model and outputs line data in which a plurality of processing results are arranged; and a readout control unit that generates a readout control signal based on the line data, the readout control signal indicating a readout target pixel group in the pixel array unit.

2. The optical detection device according to claim 1, wherein the line data includes first line data and second line data, the neural network circuit outputs the first line data and the second line data in parallel, and the readout control unit compares the first line data with the second line data and generates the readout control signal based on a comparison result.

3. The optical detection device according to claim 2, wherein the neural network model includes a spiking neural network model, each of the first line data and the second line data includes a plurality of bit strings, and each of the plurality of bit strings includes a plurality of bits representing detection results of pulses in chronological order.

4. The optical detection device according to claim 2, wherein the neural network model includes a spiking neural network model, and each of the first line data and the second line data includes a plurality of bit strings representing state values of membrane potentials in chronological order.

5. The optical detection device according to claim 1, further comprising: a conversion unit that converts at least one of identification information and sensor data and provides the converted identification information and / or sensor data to the neural network circuit, wherein the sensor outputs the identification information together with the sensor data.

6. The optical detection device according to claim 1, further comprising: a first FIFO (First-In-First-Out) memory that holds the sensor data in a first-in-first-out manner; and a second FIFO memory that holds the readout control signal in the first-in-first-out manner, wherein the neural network circuit reads the sensor data from the first FIFO memory, and the sensor reads the readout control signal from the second FIFO memory.

7. The optical detection device according to claim 1, further comprising: a first FIFO memory that holds sensor data in a first-in-first-out manner; and a second FIFO memory that holds the line data in the first-in-first-out manner, wherein the neural network circuit reads the sensor data from the first FIFO memory, and the readout control unit reads the line data from the second FIFO memory.

8. The optical detection device according to claim 7, further comprising: a digital processing unit that reads the sensor data from the first FIFO memory and processes the sensor data; and A first format processing unit that generates a communication frame storing the sensor data.

9. The optical detection device according to claim 8, further comprising: A second format processing unit that generates a communication frame storing the line data, wherein The readout control unit outputs the line data to the second format processing unit.

10. The optical detection device according to claim 1, wherein The sensor includes an EVS (event-based vision sensor).

11. The optical detection device according to claim 1, wherein The sensor includes a photon measurement circuit that counts photons.

12. The optical detection device according to claim 1, wherein The sensor includes a CIS (CMOS image sensor).

13. The optical detection device according to claim 1, wherein The sensor, the neural network circuit, and the readout control unit are dispersedly arranged on a plurality of stacked chips.

14. A control method for an optical detection device, the control method comprising: A process of reading sensor data in which a plurality of pixel data are arranged from a pixel array unit through a sensor; A process of processing the sensor data based on a neural network model and outputting line data in which a plurality of processing results are arranged; And A readout control process of generating a readout control signal based on the line data, the readout control signal indicating a readout target pixel group in the pixel array unit.

Images