Light detection device and processing device

Abstract

A light detection device includes a sensor, a neural network circuit, and a communication interface. The sensor is configured to output a plurality of first row data including a plurality of pixel data. The neural network circuit is configured to process at least one first row of data among the plurality of first row of data and output a second row of data including a result of the processing. The communication interface is configured to transmit a communication frame including at least one first line data among the plurality of first line data and the second line data, and includes: a first header including identification information of the at least one first line data among the plurality of first line data; the at least one piece of first line data in the plurality of pieces of first line data; a second header including identification information of the second row data; and the second row of data.

Description

Technical Field

[0001] The present technology relates to a light detection device, and more particularly to a light detection device and a processing device. Background Art

[0002] In recent years, signal processing using neural network models has become increasingly powerful and has been applied in a wider range of applications. For example, a device has been proposed in which a plurality of pixels and neural network circuits are pre-divided into a plurality of groups, and pixel information about each image group is input into the corresponding neural network circuit group (for example, see Patent Document 1). [Citation List] [Patent Document]

[0003] [Patent Document 1] JP 2022-525794 Summary of the Invention [Technical Issues]

[0004] The above-mentioned technology attempts to increase the processing speed by inputting pixel information about each image group into the corresponding neural network circuit group. However, in the above-mentioned device, there is a risk that it becomes difficult to synchronize the output of the sensor and the output of the neural network circuit when processing them. For example, an event-based vision sensor (EVS) using an arbitrator scheme does not sense and output address events synchronously with timing signals such as a vertical synchronization signal. In addition, for example, a spike neural network circuit also outputs spike signals asynchronously. When using these sensors or neural network circuits, there is a problem that it becomes difficult to synchronize the outputs of these sensors and circuits.

[0005] The present technology was developed in view of this situation and it is desired to make it easier to synchronize the output of the sensor and the output of the neural network circuit in a light sensing device using a neural network circuit. [Solution to the problem]

[0006] According to an embodiment of the present disclosure, the light detection device includes a sensor, a neural network circuit, and a communication interface, wherein the sensor is configured to output a plurality of first-row data, the first-row data including a plurality of pixel data. The neural network circuit is configured to process at least one first-row data among the plurality of first-row data, and output a second-row data including a result of the processing. The communication interface is configured to send a communication frame including the at least one first-row data among the plurality of first-row data and the second-row data. The communication interface includes: a first header including identification information of the at least one first-row data among the plurality of first-row data; the at least one first-row data among the plurality of first-row data; a second header including identification information of the second-row data; and the second-row data.

[0007] The processing device according to the embodiment of the present disclosure includes a circuit and a neural network circuit. The circuit is constructed as follows A communication frame is received from a light detection device. The light detection device includes: a sensor configured to output a plurality of first-line data, wherein the first-line data includes a plurality of pixel data. The neural network circuit is configured to process at least one first-line data among the plurality of first-line data and output second-line data including the result of the processing. The communication frame includes: a first header including identification information of the at least one first-line data among the plurality of first-line data; the at least one first-line data among the plurality of first-line data; a second header including identification information of the second-line data; and the second-line data. [Beneficial Effects of the Invention]

[0008] Since the format processing section generates a communication frame in which the PL and SL associated with each other are stored, downstream circuits can easily synchronize them. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram illustrating a configuration example of a light sensing device according to a first embodiment of the present technology. Figure 2 is a block diagram showing a configuration example of a sensor chip according to the first embodiment of the present technology. Figure 3 is a block diagram illustrating a configuration example of an event-based vision sensor (EVS) according to the first embodiment of the present technology. Figure 4 is a circuit diagram showing a configuration example of a pixel according to the first embodiment of the present technology. Figure 5 A and Figure 5 B is a block diagram showing a configuration example of a simulated neural network (SNN) circuit according to the first embodiment of the present technology. Figure 6 is a diagram showing an implementation example of an SNN circuit according to the first embodiment of the present technology. Figure 7 It is a block diagram showing a configuration example of the core according to the first embodiment of the present technology. Figure 8 : is a block diagram showing a configuration example of a test pattern generation section according to the first embodiment of the present technology. Figure 9 It is a block diagram showing a configuration example of a format processing section according to the first embodiment of the present technology. Figure 10 A and Figure 10 B is a diagram showing an example of a pixel row and a spike row with a header according to the first embodiment of the present technology. Figure 11 : is a diagram showing an example of output timing of a pixel row and a spike row according to the first embodiment of the present technology. Figure 12 is a diagram illustrating an example of rearranged pixel rows and spike rows according to the first embodiment of the present technology. Figure 13 is a diagram showing an example of a communication frame format according to the first embodiment of the present technology. Figure 14 is a flowchart illustrating an operation example of the light sensing device according to the first embodiment of the present technology. Figure 15 : is a diagram showing an example of a communication frame format according to a first modification example of the first embodiment of the present technology. Figure 16 : is a diagram showing an example of a communication frame format according to a second modification example of the first embodiment of the present technology. Figure 17 is a diagram illustrating an example of a multilayer structure of a sensor chip according to a third modification example of the first embodiment of the present technology. Figure 18 is a circuit diagram illustrating a configuration example of a pixel according to a third modification example of the first embodiment of the present technology. Figure 19 is a diagram illustrating an example of a multilayer structure of a sensor chip according to a fourth modification example of the first embodiment of the present technology. Figure 20 is a block diagram illustrating a configuration example of a sensor chip according to a second embodiment of the present technology. Figure 21 is a block diagram illustrating a configuration example of a sensor chip according to a first modification example of the second embodiment of the present technology. Figure 22 is a block diagram illustrating a configuration example of a sensor chip according to a second modification example of the second embodiment of the present technology. Figure 23 is a block diagram showing a configuration example of a sensor chip according to a third embodiment of the present technology. Figure 24 is a block diagram showing a configuration example of a photon measuring circuit according to a third embodiment of the present technology. Figure 25 is a circuit diagram showing a configuration example of a pixel according to a third embodiment of the present technology. Figure 26 is a block diagram showing a configuration example of a sensor chip according to a fourth embodiment of the present technology. Figure 27 is a block diagram illustrating a configuration example of a contact image sensor (CIS) according to a fourth embodiment of the present technology. Figure 28 is a circuit diagram illustrating a configuration example of a pixel according to a fourth embodiment of the present technology. Figure 29 is a block diagram showing a configuration example of a light sensing device according to a fifth embodiment of the present technology. Figure 30 is a block diagram showing a configuration example of a sensor chip according to a fifth embodiment of the present technology. Figure 31 is a diagram showing an example of a communication frame format according to a fifth embodiment of the present technology. Figure 32 is a block diagram showing an example of a schematic configuration of a vehicle control system. Figure 33 1 is a diagram showing an example of an installation position of an imaging section. DETAILED DESCRIPTION

[0010] Hereinafter, a mode for implementing the present technology (hereinafter, referred to as an embodiment) will be described. The description will be made in the following order. 1. First Embodiment (Example of Storing Pixel Rows and Spike Rows in a Communication Frame) 2. Second Embodiment (Example of Storing Pixel Rows and Spike Rows in a Communication Frame and Eliminating Circuits) 3. Third Embodiment (Example of Storing Pixel Rows and Spike Rows in a Communication Frame and Using a Photon Measurement Circuit) 4. Fourth Embodiment (Example of Storing Pixel Rows and Spike Rows in a Communication Frame and Using CIS) 5. Fifth Embodiment (Example of Storing a Peak Row in a Communication Frame) 6. Application Examples of Mobile Objects

[0011] <1. First Implementation Plan> "Example of structure of optical sensing device" Figure 1 1 is a block diagram illustrating a configuration example of a light sensing device 100 according to a first embodiment of the present technology. The light sensing device 100 includes an optical section 110, a sensor chip 200, and a DSP (digital signal processing) circuit 120. Furthermore, the light sensing device 100 includes a display section 130, an operation section 140, a bus 150, a frame memory 160, a storage section 170, and a power supply section 180. In addition to digital cameras such as digital still cameras, conceivable examples of the light sensing device 100 include, for example, smartphones, personal computers, and in-vehicle cameras.

[0012] The optical section 110 collects light from the subject and guides the light to the sensor chip 200 . The sensor chip 200 generates and processes a plurality of pieces of pixel data through photoelectric conversion. The sensor chip 200 supplies the pixel data processed into image data 209 to the DSP circuit 120 .

[0013] The DSP circuit 120 performs predetermined signal processing on the image data 209 from the sensor chip 200. The DSP circuit 120 outputs the processed image data 209 to the frame memory 160 or the like via the bus 150. According to one embodiment of the present disclosure, the predetermined signal processing includes neural network signal processing using a neural network circuit as discussed in more detail below.

[0014] The display section 130 displays image data, etc. Conceivable examples of the display section 130 include, for example, a liquid crystal panel and an organic electroluminescent (EL) panel. The operation section 140 generates an operation signal according to a user operation.

[0015] The bus 150 is a common path through which the optical section 110 , the sensor chip 200 , the DSP circuit 120 , the display section 130 , the operation section 140 , the frame memory 160 , the storage section 170 , and the power supply section 180 exchange data with one another.

[0016] The storage section 170 stores various types of data such as image data, etc. The power supply section 180 supplies power to the sensor chip 200 , the DSP circuit 120 , the display section 130 , and the like.

[0017] "Sensor Chip Structure Example" Figure 2 This is a block diagram illustrating a configuration example of a sensor chip 200 according to a first embodiment of the present technology. The sensor chip 200 is a single semiconductor chip and includes an event-based vision sensor (EVS) 300, a simulated neural network (SNN) circuit 500, header adding sections 211 and 212, and a register 213. Furthermore, the sensor chip 200 includes first-in, first-out (FIFO) memories 221 and 222, test pattern generating sections 230 and 240, digital processing sections 251 and 252, a format processing section 260, and an external communication interface 270.

[0018] The EVS 300 senses changes in the brightness of each pixel. In synchronization with the vertical synchronization signal (VSYNC), the EVS 300 internally generates a horizontal synchronization signal (HSYNC) with a higher frequency, and supplies the horizontal synchronization signal HSYNC to the header adding sections 211 and 212. In addition, in synchronization with the horizontal synchronization signal HSYNC, the EVS 300 sequentially selects multiple rows and reads out row data obtained by arranging multiple pixel data of pixels in each row as pixel rows (PL). The EVS 300 then outputs each PL to the header adding section 211. For example, each piece of pixel data includes a bit representing a sensing result of a pixel brightness change. Note that the EVS 300 is an example of a sensor described in the claims. In addition, PL is an example of the first row of data described in the claims.

[0019] The header adding section 211 generates a header including time information indicating the time of output of the PL and adds this time information to the PL. The vertical synchronization signal VSYNC and the horizontal synchronization signal HSYNC are input to the header adding section 211 as synchronization signals. For example, within the 1-V period indicated by the vertical synchronization signal VSYNC, in synchronization with the horizontal synchronization signal HSYNC, the header adding section 211 generates time information regarding the output time indicated by the signals, stores this time information in the header, and adds the header to the PL. Hereinafter, the PL to which the header is added is referred to as "PL'." The header adding section 211 supplies the PL' with the header to the FIFO memory 221.

[0020] Note that the header adding section 211 is an example of a first header adding section described in the claims, and the header added to the PL is an example of a first header described in the claims.

[0021] The FIFO memory 221 holds the PL' from the header adding section 211 by a (FIFO) scheme. The PL' with a header is read out by the test pattern generating section 230. In addition, the PL without a header is read out by the SNN circuit 500.

[0022] The SNN circuit 500 processes two or more PLs based on the SNN model and generates line data obtained by sequentially arranging the processing results as a spike line (SL). The SL is data consisting of bits arranged in time sequence, each bit indicating whether a spike occurred within a predetermined period. The SNN circuit 500 outputs the SL to the header adding unit 212. The ratio between the output frequency of the EVS 300 and the output frequency of the SNN circuit 500 is controlled to a constant value (e.g., 8:1).

[0023] Note that a neural network (NN) circuit other than the SNN circuit may be used instead of the SNN circuit 500. The SNN circuit 500 is an example of a neural network circuit described in the claims. SL is an example of the second row data described in the claims.

[0024] The header adding section 212 generates a header including time information and adds the header to the SL. The vertical synchronization signal VSYNC and the horizontal synchronization signal HSYNC are also input to the header adding section 212 as synchronization signals. Since the SNN circuit 500 generates SL from more than two PLs, the output time of the SL is delayed relative to the output time of the head of the PL group. It is assumed that the delay time Tdelay is predetermined by calculation or measurement and is stored in the register 213. The header adding section 212 reads the delay time Tdelay from the register 213 and obtains the time before the current time represented by the horizontal synchronization signal HSYNC within the 1-V period. The header adding section 212 stores time information about the time in the header and adds the header to the SL. In the following, the SL to which the header has been added is written as "SL'". The header adding section 212 supplies the SL' with the header to the FIFO memory 222.

[0025] Through the above process, the same time information is stored in the header added to the SL and the header added to the beginning of the PL group corresponding to the SL. Therefore, by referring to the header, the downstream circuit can easily synchronize the PL group and the SL.

[0026] Note that the header adding section 212 is an example of a second header adding section described in the claims, and the header added to the SL is an example of a second header described in the claims.

[0027] The FIFO memory 222 holds the SL′ from the header adding section 212 by a FIFO scheme. The SL′ with the header is read out by the test pattern generating section 240 .

[0028] The test pattern generating section 230 generates a predetermined test pattern in the test mode and supplies the test pattern to the digital processing section 251 in the test mode, and supplies PL′ to the digital processing section 251 in the non-test mode.

[0029] The test pattern generating section 240 generates a predetermined test pattern in the test mode and supplies the test pattern to the digital processing section 252 in the test mode, and supplies SL′ to the digital processing section 252 in the non-test mode.

[0030] Note that the test pattern generating sections 230 and 240 are arranged as needed. In the case where these are not necessary, PL' and SL' from the FIFO memories 221 and 222, respectively, are directly input to the digital processing sections 251 and 252, respectively.

[0031] The digital processing section 251 performs various types of digital processing on the PL′ and supplies the processed PL′ to the format processing section 260 .

[0032] The digital processing unit 252 performs various types of digital processing on the SL' as needed. The digital processing unit 252 supplies the processed SL' to the format processing unit 260. For example, the digital processing unit 252 can count the number of spikes in each SL and compare the count value with a threshold value to obtain a classification value or regression value as the recognition result of the SNN circuit 500. The processing result is output to the format processing unit 260 as needed.

[0033] The format processing section 260 generates a communication frame in which the SL′ and PL′ associated with each other are stored, and supplies the generated communication frame to the external communication interface 270 .

[0034] The external communication interface 270 transmits the communication frame from the format processing section 260 to the DSP circuit 120 and the like. As a communication standard of the external communication interface 270 , for example, the Mobile Industry Processor Interface (MIPI) is used.

[0035] "EVS Construction Example" Figure 3 3 is a block diagram showing a configuration example of an EVS 300 according to a first embodiment of the present technology. The EVS 300 includes a drive section 310, a pixel array section 320, a timing control circuit 330, and a row scanner 340. A plurality of pixels 400 are arranged in a two-dimensional grid in the pixel array section 320.

[0036] The driving section 310 drives each pixel 400. The pixel 400 senses whether there is a change in brightness and generates pixel data indicating the sensing result.

[0037] The timing control circuit 330 controls the driving timing of the driving section 310 and the line scanner 340. A vertical synchronization signal VSYNC is input to the timing control circuit 330. The timing control circuit 330 generates a horizontal synchronization signal HSYNC based on the vertical synchronization signal VSYNC and supplies the horizontal synchronization signal HSYNC to the line scanner 340 and the header adding sections 211 and 212.

[0038] In synchronization with the horizontal synchronization signal HSYNC, the row scanner 340 sequentially selects rows (rows, columns, etc.) and reads out the pixel data of each pixel in each selected row. The row scanner 340 arranges the pixel data read out from a row in one dimension and outputs the data as PL to the header adding unit 211. Note that the readout operation is assumed to be performed in units of rows, but the readout operation can also be performed in units of regions. In this case, the row scanner 340 arranges the individual pieces of pixel data read out from the selected region in a one-dimensional manner in a predetermined order and outputs the pixel data as PL.

[0039] As shown in the figure, the control of sequentially reading out pixel data in units of rows or regions in synchronization with a synchronization signal such as the HSYNC signal is called a scanning scheme. Note that as described later, the EVS 300 can also use an arbiter scheme that reads out pixel data without synchronization with a synchronization signal.

[0040] "Pixel Structure Example" Figure 4 4 is a circuit diagram showing a configuration example of a pixel 400 according to the first embodiment of the present technology. Each pixel 400 includes a pixel circuit 410 , a buffer 420 , a differentiating circuit 430 , and a quantizer 440 .

[0041] The pixel circuit 410 includes a photodiode 411 , negative-channel metal oxide semiconductor (nMOS) transistors 412 and 413 , and a positive-channel (pMOS) transistor 414 .

[0042] Photodiode 411 generates photocurrent by photoelectrically converting incident light. An nMOS transistor 412 is interposed between the power supply and photodiode 411. A pMOS transistor 414 and an nMOS transistor 413 are connected in series between the power supply and ground. Furthermore, the gate of nMOS transistor 413 is connected between nMOS transistor 412 and photodiode 411. A bias voltage Vblog is applied to the gate of pMOS transistor 414.

[0043] Buffer 420 includes pMOS transistors 421 and 422 connected in series between a power supply and a ground terminal. The gate of ground-side pMOS transistor 422 is connected between pMOS transistor 414 and nMOS transistor 413. Bias voltage Vbsf is applied to the gate of power-side pMOS transistor 421. Differentiating circuit 403 is connected between pMOS transistors 421 and 422.

[0044] Using the above circuit, a voltage signal according to the photocurrent is generated and output from the buffer 420 .

[0045] The differentiating circuit 430 includes capacitors 431 and 433 , pMOS transistors 432 and 434 , and an nMOS transistor 435 .

[0046] One end of capacitor 431 is connected to buffer 420, and the other end of capacitor 431 is connected to one end of capacitor 433 and the gate of pMOS transistor 434. Reset signal xrst is input to the gate of pMOS transistor 432, and the source and drain are connected to both ends of capacitor 433. PMOS transistor 434 and nMOS transistor 435 are connected in series between the power supply and the ground terminal. In addition, the other end of capacitor 433 is connected between pMOS transistor 434 and nMOS transistor 435. A bias voltage Vba is applied to the gate of ground-side nMOS transistor 435, and quantizer 440 is connected between pMOS transistor 434 and nMOS transistor 435. With this connection, a differential signal representing the amount of change in the voltage signal is generated and output to quantizer 440. In addition, the differential signal is initialized by reset signal xrst.

[0047] Quantizer 440 includes a pMOS transistor 441 and an nMOS transistor 442 connected in series between a power supply and a ground terminal. The gate of pMOS transistor 441 is connected to differentiating circuit 430, and a predetermined upper threshold value Vbon is applied to the gate of nMOS transistor 442. The voltage signal between pMOS transistor 441 and nMOS transistor 442 is read by row scanner 340 as a brightness change sensing signal.

[0048] In this figure, when the differential signal representing the brightness change exceeds the upper threshold value Vbon, an ON event is sensed. Note that when the differential signal drops below the lower threshold value Vboff, the pixel 400 can also sense an OFF event. In this case, a pMOS transistor 443 and an nMOS transistor 444 connected in series between the power supply and the ground terminal are added. The gate of the pMOS transistor 443 is connected to the differentiating circuit 430, and the lower threshold value Vboff is applied to the gate of the nMOS transistor 444. The pixel 400 can sense both ON and OFF events, or can sense only one of them.

[0049] "Example of SNN Circuit Structure" Figure 5 A is a block diagram showing a configuration example of an SNN circuit 500. As shown in the figure, the SNN circuit 500 includes an input layer 511, an intermediate layer 512, and an output layer 513.

[0050] The PL is input to the input layer 511. One or more layers are arranged in the intermediate layer 512. Neurons in the upper layer connect to neurons in the lower layer, and the calculation results of the upper layer are transmitted to the lower layer. The output layer 513 asynchronously generates spike signals. The spike signals generated within a predetermined period are arranged in time sequence and output as a SL. A predetermined number of two or more PLs are sequentially input to the SNN circuit 500 to generate a SL.

[0051] Note that, as described below Figure 5 As shown in B, the output layer 513 includes multiple neurons and can also output multiple SLs in parallel.

[0052] In addition, the data size output by the SNN circuit 500 can be changed according to the network settings. Figure 5 When outputting multidimensional data as shown in B, the data can be output sequentially by dividing the data into one-dimensional rows.

[0053] Figure 6 is a diagram showing an implementation example of the SNN circuit 500 according to the first embodiment of the present technology. For example, Figure 5 The SNN circuit 500 in Figure 6 The circuit implementation in . Figure 6 As shown, for example, the SNN circuit 500 includes an input / output (I / F) interface 520 and a multi-core array 530 .

[0054] The I / F interface 520 performs data transmission and reception between the outside and the multi-core array 530. The I / F interface 520 supplies the multi-core array with PL input from the FIFO memory 221, and supplies the header adding section 212 with SL from the multi-core array 530.

[0055] The multi-core array 530 includes a plurality of cores 550 arranged in a two-dimensional grid. A router 540 is disposed adjacent to each core 550.

[0056] Routers 540 control data paths. For example, each router 540 includes FIFO memories 541 to 545 and an arbiter 546. "E" in the figure indicates the eastward direction from the router 540 of interest, and "S" indicates the southward direction from the router 540 of interest. "W" indicates the westward direction, and "N" indicates the northward direction. "L" indicates the direction toward the core 550 adjacent to the router 540.

[0057] FIFO memory 541 stores data from the east direction using a FIFO scheme and outputs a request to arbiter 546. FIFO memory 542 stores data from the south direction using a FIFO scheme and outputs a request to arbiter 546. FIFO memory 543 stores data from the west direction using a FIFO scheme and outputs a request to arbiter 546. FIFO memory 544 stores data from the north direction using a FIFO scheme and outputs a request to arbiter 546. FIFO memory 545 stores data from adjacent core 550 using a FIFO scheme and outputs a request to arbiter 546.

[0058] Note that the external FIFO memory 221 may also be eliminated, and the FIFO memory 541 or the like in the SNN circuit 500 may serve as a substitute.

[0059] The arbiter 546 arbitrates between requests from the FIFO memories 541 to 545 and returns a response. After receiving the response, the FIFO memory outputs data to any core 550 located in the east, west, south, and north and adjacent to the FIFO memory via the arbiter 546.

[0060] Figure 7 1 is a block diagram showing a configuration example of a core 550 according to the first embodiment of the present technology. The core 550 includes a core router 551, a neuron I / O 552, a product-sum unit 553, a working memory 554, a membrane potential memory 555, and a leaky integrate and fire (LIF) unit 556.

[0061] The core router 551 supplies data from the adjacent router 540 to the neuron I / O 552 , and supplies data from the LIF unit 556 to the adjacent router 540 .

[0062] The product-sum unit 553 integrates the data from the neuron I / O 552 using the working memory 554. The membrane potential memory 555 stores the membrane potential obtained by the integration. The LIF unit 556 determines whether the membrane potential has exceeded a predetermined threshold and is triggered (i.e., whether a spike has occurred), and supplies the determination result to the core router 551.

[0063] "Configuration Example of Test Pattern Generator" Figure 8 2 is a block diagram showing a configuration example of the test pattern generating section 230 according to the first embodiment of the present technology. The test pattern generating section 230 includes a test pattern supply section 231 and a switch 232 .

[0064] When the test mode is started due to the control signal MODE, the test pattern supply section 231 generates a predetermined test pattern and supplies the predetermined test pattern to the switch 232 .

[0065] When the test mode has started, the switch 232 supplies the test pattern to the digital processing section 251, and when the non-test mode has started, supplies PL' from the FIFO memory 221 to the digital processing section 251. Note that the configuration of the test pattern generating section 240 is similar to that of the test pattern generating section 230.

[0066] "Format Processing Unit Structure Example" Figure 9 260 is a block diagram showing a configuration example of the format processing section 260 according to the first embodiment of the present technology. The format processing section 260 includes a buffer memory 261 , a rearrangement processing section 262 , and a formatter 263 .

[0067] The buffer memory 261 temporarily holds PL′ and SL′ from the digital processing sections 251 and 252 , respectively.

[0068] The rearrangement processing unit 262 reads out PL' and SL' from the buffer memory 261 and rearranges their arrays based on the delay time held in the register 213. The rearrangement process will be described in detail later. The rearrangement processing unit 262 supplies the rearranged PL' and SL' to the formatter 263.

[0069] The formatter 263 generates a communication frame in a format conforming to a predetermined communication standard, adds a trailer to each of PL′ and SL′ having a header, stores data in the communication frame, and supplies the communication frame to the external communication interface 270 .

[0070] Figure 10 A and Figure 10 B is a diagram showing an example of PL' and SL' with a header according to the first embodiment of the present technology. Figure 10 A shows an example of PL', and Figure 10 B shows an example of SL'.

[0071] like Figure 10 As shown in FIG. 1A , PL′ includes a pixel header (PH) and PL. For example, PH stores time information therein. PL includes multiple pieces of pixel data. For example, each piece of pixel data is a 1-bit information indicating whether an ON event has been sensed. x0 to xi in the figure represent the x-coordinates of the pixels in the row to which the y-coordinates are assigned. Note that in the case of sensing both an ON event and an OFF event, two bits of information are stored for each pixel.

[0072] like Figure 10As shown in B, SL' includes a spike header (SH) and SL. For example, SH stores time information therein. SL has a plurality of spike signals arranged in time sequence that are output within a predetermined period. Each spike signal is 1-bit information indicating whether a spike occurs. t0 to tj in the figure indicate the time when the spike signal is output. Note that the SNN circuit 500 can also output the value of the membrane potential in time sequence instead of SL. In this case, row data obtained by arranging two or more bits of digital values representing the membrane potential in time sequence is output.

[0073] Note that the header adding sections 211 and 212 store time information in the header, but this configuration is not the only example. For example, the header adding sections 211 and 212 may store identification information (row number, etc.) of the corresponding row in the header instead of time information.

[0074] Figure 11 1 is a diagram showing an example of output timing of a pixel row and a spike row according to the first embodiment of the present technology. Assume that one piece of image data includes PL1 to PLk.

[0075] In the figure, first image data PLs1 is output from the EVS 300 at or after time T1p. Second image data PLs2 is output at or after time T2p after a predetermined blanking period. Then, third image data PLs3 is output at or after time T3p after a predetermined blanking period, and fourth image data PLs4 is output at or after time T4p.

[0076] On the other hand, the SNN circuit 500 sequentially outputs SL1 to SLn at or after time T1s after processing PLs1 and while outputting PLs2. As described above, this SL group is referred to as SLs1 and SLs2. Then, the SNN circuit 500 sequentially outputs SL1 to SLn of SLs2 at or after time T2s after processing PLs3 and while outputting PLs4. This SL group is referred to as SLs2. Since the EVS 300 and the SNN circuit 500 operate in parallel, in some cases, as shown in the figure, the SNN circuit 50 continues to output SLs during the blanking period.

[0077] As shown in the figure, the output timing of the SL corresponding to the PL group is delayed relative to the output timing of the first PL group. For example, the difference between the output timing T1p of PL1 at the beginning of PLs1 and the output timing Tls of SL1 at the beginning of SLs1 corresponding to the PL group corresponds to the delay time Tdelay of SL1. Register 213 stores this delay time Tdelay.

[0078] The rearrangement processing unit 262 rearranges the arrays of PL and SL to facilitate synchronization based on the delay time in the register 213. For example, since SLs1 is generated from PLs1, the corresponding SLs1 is arranged after PLs1.

[0079] Figure 12 This diagram shows an example of rearranged pixel rows and spike rows according to the first embodiment of the present technology. As shown in the figure, SLs1 is arranged after PLs1. Next, PLs2 and PLs3 are arranged, and SLs2 corresponding to PLs3 is arranged. Then, PLs4 is arranged.

[0080] As shown in the figure, by arranging the PL and the SL corresponding to the PL adjacent to each other, the downstream circuit can easily synchronize them.

[0081] Figure 13 This figure shows an example of a communication frame format according to the first embodiment of the present technology. A communication frame stores a frame header, a destination address, a sender address, and data. PL1 to PLk are sequentially stored in the data. The header adding unit 211 adds a PH to each PL. Furthermore, the formatter 263 adds a pixel footer (PF) to each PL.

[0082] Then, after PLk, SL1 to SLn are sequentially stored. The header adding unit 212 adds SH to each SL. In addition, the formatter 263 adds a spike footer (SF) to each SL.

[0083] In addition, in order to adjust the data size, the formatter 263 inserts padding data as needed.

[0084] As shown in the figure, the format processing section 260 generates a communication frame in which PL and SL associated with each other are stored. Therefore, downstream circuits can easily synchronize PL and SL.

[0085] "Operation Example of Light Sensing Device" Figure 14 1 is a flowchart showing an operation example of the light sensing device 100 according to the first embodiment of the present technology. For example, when a predetermined application for capturing image data is executed, the operation is started.

[0086] The EVS 300 and the SNN circuit 500 generate the PL and SL, respectively (step S901). Furthermore, the header adding units 211 and 212 add headers to the PL and SL, respectively (step S902). Furthermore, the digital processing units 251 and 252 perform digital processing on the PL' and SL' with the headers, respectively (step S903). Furthermore, the format processing unit 260 generates a communication frame through format processing (step S904), and the external communication interface 270 transmits the communication frame externally (step S905). After step S905, step S901 and subsequent steps are repeatedly executed.

[0087] In this way, according to the first embodiment of the present technology, since the format processing section 260 generates a communication frame in which the PL and SL associated with each other are stored, the downstream circuit can easily synchronize them.

[0088] "First Modification" Although the header adding sections 211 and 212 add a header for each PL or each SL according to the first embodiment, this format is not the only example. The light sensing device 100 according to the first modification of the first embodiment differs from the first embodiment in that a header is added only at the beginning of each of the PL group and the SL group.

[0089] Figure 15 This figure illustrates an example of a communication frame format according to a first variation of the first embodiment of the present technology. According to the first variation of the first embodiment, the header adding unit 211 adds a header information field (Phase Format) only to the header PL1 at the beginning of each PL group (PL1 to PLk) included in the image data. For example, the header information field stores timing information regarding the output of the PL at the beginning of each PL group (PL1 to PLk) or identification information regarding the row or frame at the beginning. Furthermore, the formatter 263 adds a packet forwarding function (PF) only to the header PLk at the end of each PL group (PL1 to PLk).

[0090] The header adding unit 212 adds SH only to SL1 at the beginning of the SL group (SL1 to SLn) corresponding to the PL group (PL1 to PLk). The formatter 263 adds SF only to SLn at the end of the SL group (SL1 to SLn).

[0091] As shown in the figure, by adding a header to the beginning of each PL group and each SL group, the processing load of the header adding units 211 and 212 can be reduced, compared to the case where a header is added to each PL group and each SL group separately. In addition, by adding a trailer to the end of each PL group and each SL group, the processing load of the formatter 263 can be reduced, compared to the case where a trailer is added to each PL group and each SL group. In addition, the data size of the PL group and the SL group can be reduced.

[0092] In this way, according to the first modification example of the first embodiment of the present technology, since the header adding section 211 or 212 adds a header to the beginning of each of the PL group and the SL group, the processing amount and the data size can be reduced.

[0093] "Second Modification" Although the EVS 300 uses the scanning scheme synchronized with the synchronization signal according to the first embodiment, an asynchronous arbiter scheme may be used instead. The light sensing device 100 according to the second modification of the first embodiment differs from the first embodiment in that an arbiter scheme is used.

[0094] Figure 16 This figure illustrates an example of a communication frame format according to a second variation of the first embodiment of the present technology. According to the second variation of the first embodiment, the EVS 300 generates and outputs PLs using an arbiter scheme that is not synchronized with a synchronization signal. PLs 1 to PLk are output not in a fixed order but in the order in which address events are sensed.

[0095] In addition, for each pixel in a row, the PL includes information about the pixel's x-coordinate and time. The time information indicates the time when the address event was sensed. For example, in the figure, "x7, t0" indicates that the address event was sensed at the pixel at coordinate x7 at time t0.

[0096] Since time information is included in PL in the case of the arbiter scheme, the header adding section 211 stores identification information (row number, etc.) about the area or row in PH. In addition, the header adding section 212 also stores the identification information in SH.

[0097] In this way, according to the second modification of the first embodiment of the present technology, since the header adding sections 211 and 212 store the line number and the like in the header, synchronization between the PL and the SL can be maintained even when the arbiter scheme is used.

[0098] "Third Modification" Although circuits such as the EVS 300 are arranged on a single semiconductor chip according to the first embodiment described above, this configuration may cause difficulties in increasing the number of pixels in some cases. The light sensing device 100 according to the third modification of the first embodiment differs from the first embodiment in that the circuits are arranged in a distributed manner on two stacked semiconductor chips.

[0099] Figure 17This figure illustrates an example of a multilayer structure of a sensor chip 200 according to a third modification of the first embodiment of the present technology. The sensor chip 200 according to the third modification of the first embodiment includes a pixel chip 201 and a circuit chip 202. These chips are stacked on top of each other and electrically connected, for example, via copper-copper (Cu-Cu) bonding. Note that in addition to Cu-Cu bonding, they can also be connected via vias or bumps.

[0100] Figure 18 This is a circuit diagram showing a configuration example of a pixel 400 according to a third modification of the first embodiment of the present technology. For example, the pixel circuit 410 in the pixel 400 is arranged on the pixel chip 201, and downstream circuits such as the buffer 420 and subsequent circuits are arranged on the circuit chip 202.

[0101] Note that the circuits to be arranged on each chip are not limited to those shown in the figure. For example, the photodiode 411 and nMOS transistors 412 and 413 may be arranged on the pixel chip 201, and the remaining circuits may be arranged on the circuit chip 202. Alternatively, only the photodiode 411 may be arranged on the pixel chip 201, and the remaining circuits may be arranged on the circuit chip 202.

[0102] Note that each of the first and second modifications can be applied to the third modification of the first embodiment.

[0103] In this manner, according to the third modification of the first embodiment of the present technology, since the circuits are arranged in a distributed manner on the two stacked chips, the circuit scale of each chip can be reduced. Therefore, it becomes easier to increase the number of pixels.

[0104] "Fourth Modification" Although circuits such as the EVS 300 are arranged on a single semiconductor chip according to the first embodiment described above, this configuration may cause difficulties in increasing the number of pixels in some cases. The light sensing device 100 according to the fourth modification of the first embodiment differs from the first embodiment in that the circuits are arranged in a distributed manner on three stacked semiconductor chips.

[0105] Figure 19This figure shows an example of a multilayer structure of a sensor chip 200 according to a fourth modification of the first embodiment of the present technology. According to the fourth modification of the first embodiment, the sensor chip 200 includes a stacked pixel chip 201, a circuit chip 202, and a circuit chip 203. Some pixels of the EVS 300 (e.g., pixel circuit 410, etc.) are arranged on the pixel chip 201, and the remaining circuits of the EVS 300 are arranged on the circuit chip 202. In addition, downstream circuits such as the header adding unit 211 and subsequent circuits are arranged on the circuit chip 203. Note that the circuits to be arranged on each chip are not limited to the circuits shown in the figure. In addition, the number of chips to be stacked is not limited to three, but may be equal to or greater than four.

[0106] Note that each of the first and second modifications can be applied to the fourth modification of the first embodiment.

[0107] In this manner, according to the fourth modification of the first embodiment of the present technology, since the circuits are arranged in a distributed manner on three stacked chips, the circuit scale of each chip can be reduced. Therefore, it becomes easier to increase the number of pixels.

[0108] <2. Second Implementation Plan> Although two FIFO memories and two digital processing units are arranged according to the first embodiment described above, these numbers can be reduced. The light sensing device 100 according to the second embodiment differs from the first embodiment in that the FIFO memory 222 and the digital processing unit 252 are eliminated.

[0109] Figure 20 2 is a block diagram showing a configuration example of a sensor chip 200 according to a second embodiment of the present technology. The sensor chip 200 according to the second embodiment differs from the first embodiment in that the FIFO memory 222, the test pattern generation unit 240, and the digital processing unit 252 are not provided.

[0110] In the second embodiment, the header adding section 212 supplies the header-containing SL' to the FIFO memory 221, which stores the PL' and SL'. Furthermore, the FIFO memory 221 supplies the PL to the SNN circuit 500, and supplies the PL' and SL' to the digital processing section 251 via the test pattern generating section 230. The digital processing section 251 processes the PL' and SL' and supplies them to the format processing section 260.

[0111] Note that each of the first, second, third, and fourth modifications of the first embodiment can be applied to the second embodiment.

[0112] In this manner, according to the second embodiment of the present technology, since the FIFO memory 222 and the digital processing section 252 are eliminated, the circuit scale of the sensor chip 200 can be reduced.

[0113] "First Modification" Although the FIFO memory 221 holds the PL and supplies it to the SNN circuit 500 according to the above-described second embodiment, the EVS 300 may output the PL directly to the SNN circuit 500. The light sensing device 100 according to the first modification of the second embodiment is different from the second embodiment in that the EVS 300 outputs the PL directly to the SNN circuit 500.

[0114] Figure 21 This is a block diagram illustrating an example configuration of a sensor chip 200 according to a first variation of the second embodiment of the present technology. According to the first variation of the second embodiment, the EVS 300 bypasses the FIFO memory 221 to output the PL to the SNN circuit 500 and the header adding unit 211. By bypassing the FIFO memory 221 and outputting the PL directly to the SNN circuit 500, the delay time between the SL and the PL can be shortened.

[0115] Note that each of the first, second, third, and fourth modifications of the first embodiment can be applied to the first modification of the second embodiment.

[0116] In this manner, according to the first modification example of the second embodiment of the present technology, since the EVS 300 directly outputs the PL to the SNN circuit 500 , the delay time of the SL can be shortened.

[0117] "Second Modification" Although the FIFO memory 221 holds PL' and SL' according to the second embodiment described above, PL' and SL' may be held in separate FIFO memories. The light sensing device 100 according to the second modification of the second embodiment differs from the first embodiment in that PL' and SL' are held in separate FIFO memories.

[0118] Figure 22 1 is a block diagram showing a configuration example of a sensor chip 200 according to a second modification of the second embodiment of the present technology. The sensor chip 200 according to the second modification of the second embodiment differs from that of the second embodiment in that a FIFO memory 222 is further provided.

[0119] In the second modification of the second embodiment, the header adding section 212 supplies the header SL′ to the FIFO memory 222. The FIFO memory 222 holds the SL′ and supplies it to the test pattern generating section 240.

[0120] In this manner, according to the second modification of the second embodiment of the present technology, since the FIFO memory 222 is added, PL′ and SL′ can be held in different FIFO memories.

[0121] <3. Third Implementation Plan> Although the EVS 300 is used as a sensor generating PL according to the first embodiment, a photon measuring circuit that counts photons may be used instead of the EVS 300. The light sensing device 100 according to the third embodiment differs from the first embodiment in that a photon measuring circuit is used instead of the EVS 300.

[0122] Figure 23 This is a block diagram illustrating a configuration example of a sensor chip 200 according to a third embodiment of the present technology. The sensor chip 200 according to the third embodiment differs from the first embodiment in that a photon measurement circuit 600 is provided in place of the EVS 300. Note that the photon measurement circuit 600 is an example of the sensor described in the claims.

[0123] Figure 24 This is a block diagram illustrating a configuration example of a photon measurement circuit 600 according to a third embodiment of the present technology. The photon measurement circuit 600 includes a drive section 610, a pixel array section 620, a timing control circuit 640, and a readout processing section 650. A plurality of pixels 630 are arranged in a two-dimensional grid in the pixel array section 620.

[0124] The functions of the driving section 610 , the pixel array section 620 , the timing control circuit 640 , and the readout processing section 650 are similar to those of the driving section 310 , the pixel array section 320 , the timing control circuit 330 , and the row scanner 340 , respectively.

[0125] Figure 25 : is a circuit diagram showing a configuration example of a pixel 630 according to a third embodiment of the present technology. Each pixel 630 includes a quenching resistor 631, a single photon avalanche diode (SPAD) 632, an inverter 633, and a photon counter 634.

[0126] The quenching resistor 631 and the SPAD 632 are connected in series. The inverter 633 inverts the voltage signal between the quenching resistor 631 and the SPAD 632 and supplies the voltage signal as a pulse signal to the photon counter 634. The photon counter 634 counts the number of pulses in the pulse signal, reads out pixel data representing the count value, and supplies the pixel data to the readout processing unit 650.

[0127] When counting photons, each piece of pixel data in the PL is a bit string of two or more bits representing a count value. However, according to the first embodiment, each piece of pixel data is preferably converted into one bit of information. When performing this conversion, for example, a conversion circuit that converts the bit string into one bit for each pixel is inserted upstream of the SNN circuit 500.

[0128] Note that the circuit configuration of the pixel 630 is not limited to the circuit configuration shown in the figure as long as it can count photons.

[0129] In addition, each of the first, second, third, and fourth modified examples of the first embodiment, the second embodiment, and the first and second modified examples of the second embodiment can be applied to the third embodiment.

[0130] In this manner, according to the third embodiment of the present technology, since the photon measurement circuit 600 is arranged instead of the EVS 300 , synchronization can be maintained between the output of the photon measurement circuit 600 and the output of the SNN circuit 500 .

[0131] <4. Fourth Implementation Plan> Although the EVS 300 is used as a sensor generating PL according to the first embodiment, a contact image sensor (CIS) may be used instead of the EVS 300. The light sensing device 100 according to the fourth embodiment differs from the first embodiment in that a CIS is used instead of the EVS 300.

[0132] Figure 26 : is a block diagram showing a configuration example of a sensor chip 200 according to a fourth embodiment of the present technology. The sensor chip 200 according to the fourth embodiment is different from the first embodiment in that a CIS 700 is arranged instead of the EVS 300. Note that the CIS 700 is an example of a sensor described in the claims.

[0133] Figure 27 7 is a block diagram illustrating a configuration example of a CIS 700 according to a fourth embodiment of the present technology. The CIS 700 includes a vertical scanning circuit 710, a timing control circuit 720, a digital-to-analog converter (DAC) 730, a pixel array section 740, a column analog-to-digital converter (ADC) 760, and a horizontal transfer scanning circuit 770. Pixels 750 are arranged in a two-dimensional grid in the pixel array section 740.

[0134] The vertical scanning circuit 710 sequentially selects and drives rows and causes the rows to output analog pixel signals to the column ADC 760. The timing control circuit 720 generates a horizontal synchronization signal HSYNC based on the vertical synchronization signal VSYNC and supplies the horizontal synchronization signal HSYNC to the horizontal transfer scanning circuit 770 and the header adding sections 211 and 212.

[0135] The DAC 730 generates a predetermined reference signal and supplies the predetermined reference signal to the column ADC 760. For example, a ramp signal of a sawtooth wave pattern is used as the reference signal.

[0136] The column ADC 760 includes an ADC for each column and performs analog-to-digital (AD) conversion on each pixel signal of the column. The column ADC 760 generates PL according to the control of the horizontal transfer scanning circuit 770 and outputs PL to the header adding section 211.

[0137] The horizontal transfer scanning circuit 770 controls the column ADC 760 to sequentially output pixel data.

[0138] As described above, in the CIS, each piece of pixel data in the PL is a bit string of two or more bits representing the grayscale value of the pixel. However, according to the first embodiment, each piece of pixel data is preferably converted into one bit of information. When performing the conversion, for example, a conversion circuit that converts the bit string into one bit for each pixel is inserted upstream of the SNN circuit 500.

[0139] Figure 28 7 is a circuit diagram showing a configuration example of a pixel 750 according to a fourth embodiment of the present technology. The pixel 750 includes a photodiode 751 , a transfer transistor 752 , a reset transistor 753 , a floating diffusion layer 754 , an amplifying transistor 755 , and a selecting transistor 756 .

[0140] The photodiode 751 performs photoelectric conversion on incident light and generates electric charge. The transfer transistor 752 transfers the electric charge from the photodiode 751 to the floating diffusion layer 754 in accordance with a transfer signal TRG from the vertical scanning circuit 710 .

[0141] The reset transistor 753 extracts charges from the floating diffusion layer 754 and initializes the floating diffusion layer 754 according to the reset signal RST from the vertical scanning circuit 710. The floating diffusion layer 754 accumulates charges and generates a voltage according to the amount of the charges.

[0142] The amplifier transistor 755 amplifies the voltage of the floating diffusion layer 754. In response to the selection signal SEL from the vertical scanning circuit 710, the selection transistor 756 outputs the amplified voltage signal as a pixel signal.

[0143] In addition, a vertical signal line 759 is placed for each column in the pixel array section 740 , and a pixel signal of each pixel 750 in a column is output to the column ADC 760 via the vertical signal line 729 of the column.

[0144] Note that the circuit configuration of the pixel 750 is not limited to the configuration shown in the figure as long as it can generate an analog pixel signal.

[0145] In addition, each of the first, second, third, and fourth modified examples of the first embodiment, the second embodiment, and the first and second modified examples of the second embodiment can be applied to the fourth embodiment.

[0146] In this manner, according to the fourth embodiment of the present technology, since the CIS 700 is arranged in place of the EVS 300 , synchronization between the output of the CIS 700 and the output of the SNN circuit 500 can be maintained.

[0147] <5. Fifth Implementation Plan> Although the sensor chip 200 transmits both PL and SL according to the first embodiment, it is conceivable that only SL is required in the downstream circuit. The light sensing device 100 according to the fifth embodiment differs from the first embodiment in that the sensor chip 200 transmits only SL of PL and SL.

[0148] Figure 29 5 is a block diagram showing a configuration example of a light sensing device 100 according to a fifth embodiment of the present technology. The light sensing device 100 includes a system global clock supply section 191 , an external sensor 192 , a sensor chip 200 , and a DSP circuit 120 .

[0149] The system global clock supply section 191 generates a global clock signal CLKg and supplies the global clock signal CLKg to the external sensor 192 and the sensor chip 200 .

[0150] The external sensor 192 operates in synchronization with the global clock signal CLKg, and transmits a communication frame in which predetermined sensor data is stored to the DSP circuit 120. Note that various types of devices may be arranged instead of the external sensor 192.

[0151] The sensor chip 200 also operates in synchronization with the global clock signal CLKg. Furthermore, the sensor chip 200 adds a header and the like to the SL, stores the SL in a communication frame, and transmits the communication frame to the DSP circuit 120. The DSP circuit 120 processes the sensor data and the SL while maintaining synchronization between the sensor data and the SL.

[0152] Figure 301 is a block diagram showing a configuration example of a sensor chip 200 according to a fifth embodiment of the present technology. The sensor chip 200 according to the fifth embodiment differs from the first embodiment in that the header adding section 211, the test pattern generating section 230, and the digital processing section 251 are not provided.

[0153] In the fifth embodiment, the EVS 300 outputs the PL to the FIFO memory 221. In addition, the format processing section 260 stores the header SL' in the communication frame.

[0154] Figure 31 1 is a diagram showing an example of a communication frame format according to a fifth embodiment of the present technology. As shown in the diagram, SL' is stored in the communication frame, but PL' is not stored. This can reduce the communication traffic of the external communication interface 270.

[0155] Note that each of the third and fourth modified examples of the multilayer structure of the first embodiment, the third embodiment using a photon measurement circuit, and the fourth embodiment using a CIS can be applied to the fifth embodiment.

[0156] In this way, according to the fifth embodiment of the present technology, since the format processing section 260 does not store PL′ in the communication frame, the communication volume of the external communication interface 270 can be reduced.

[0157] <6. Application Examples for Mobile Objects> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure can be implemented as a device installed on any type of mobile object such as an automobile, electric vehicle, hybrid electric vehicle, motorcycle, bicycle, personal mobility device, aircraft, drone, ship, or robot.

[0158] Figure 32 : is a block diagram showing an example of a schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to the embodiment of the present disclosure can be applied.

[0159] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Figure 32 In the illustrated example, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an exterior information detection unit 12030, an interior information detection unit 12040, and an integrated control unit 12050. Furthermore, a microcomputer 12051, a sound / image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown as functional components of the integrated control unit 12050.

[0160] Drive system control unit 12010 controls the operation of devices related to the vehicle's drive system according to various programs. For example, drive system control unit 12010 functions as a control device for the following devices: a drive force generating device such as an internal combustion engine or a drive motor for generating drive force for the vehicle; a drive force transmission mechanism for transmitting drive force to the wheels; a steering mechanism for adjusting the steering angle of the vehicle; and a braking device for generating braking force for the vehicle.

[0161] The body system control unit 12020 controls the operation of various devices attached to the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, power windows, and various lights such as headlights, taillights, brake lights, turn signals, and fog lights. In this case, radio waves transmitted from a portable device serving as a key substitute or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals and controls the vehicle's door locks, power windows, and lights.

[0162] The vehicle exterior information detection unit 12030 detects information about the exterior of the vehicle having the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to the imaging unit 12031. The vehicle exterior information detection unit 12030 instructs the imaging unit 12031 to provide an image of the vehicle exterior, and then receives the image from the imaging unit 12031. Based on the received image, the vehicle exterior information detection unit 12030 processes the received image to detect objects such as people, vehicles, obstacles, signs, or symbols on the road surface, or processes the received image to detect the distance to such objects.

[0163] Imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. Imaging unit 12031 can output the electrical signal as an image or as information regarding the measured distance. The light received by imaging unit 12031 can be visible light or invisible light such as infrared light.

[0164] The in-vehicle information detection unit 12040 detects information about the vehicle interior. For example, the in-vehicle information detection unit 12040 is connected to a driver status detection unit 12041 that detects the driver's condition. For example, the driver status detection unit 12041 includes a camera that captures the driver's image. Based on the detection information input from the driver status detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's fatigue level or concentration level, or determine whether the driver is dozing off.

[0165] The microcomputer 12051 can calculate control target values for the driving force generating device, steering mechanism, or braking device based on information about the interior or exterior of the vehicle (this information is obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040), and can output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform coordinated control to implement functions of an advanced driver assistance system (ADAS), including collision avoidance or impact mitigation, vehicle-to-vehicle distance-based following, vehicle speed maintenance, vehicle collision warning, or vehicle lane departure warning.

[0166] In addition, the microcomputer 12051 can perform collaborative control with the intention of achieving automatic driving (for example, operating the vehicle without input from the driver, etc.) by controlling a driving force generating device, a steering mechanism, or a braking device, etc. based on information about the interior or exterior of the vehicle (the information is obtained by the exterior information detection unit 12030 or the interior information detection unit 12040).

[0167] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 based on information about the exterior of the vehicle, which information is obtained by the exterior information detection unit 12030. For example, the microcomputer 12051 can perform cooperative control aimed at preventing glare by controlling the headlights to switch from high beam to low beam based on the position of a preceding vehicle or an oncoming vehicle detected by the exterior information detection unit 12030.

[0168] The sound / image output unit 12052 transmits an output signal of at least one of sound and image to an output device that can visually or auditorily notify the vehicle's passengers or the outside of the vehicle of information. Figure 32 In the example of FIG, an audio speaker 12061, a display portion 12062, and an instrument panel 12063 are shown as output devices. For example, the display portion 12062 may include at least one of an in-vehicle display and a head-up display.

[0169] Figure 33 12031 is a diagram showing an example of the installation position of the imaging unit 12031.

[0170] exist Figure 33 , the imaging unit 12031 includes imaging units 12101 , 12102 , 12103 , 12104 and 12105 .

[0171] Imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, on the front nose, side mirrors, rear bumper, and rear door of vehicle 12100, as well as on the upper portion of the vehicle's interior windshield. Imaging unit 12101 provided on the front nose and imaging unit 12105 provided on the upper portion of the vehicle's interior windshield primarily capture images of the front of vehicle 12100. Imaging units 12102 and 12103 provided on the side mirrors primarily capture images of the sides of vehicle 12100. Imaging unit 12104 provided on the rear bumper or rear door primarily captures images of the rear of vehicle 12100. Imaging unit 12105 provided on the upper portion of the vehicle's interior windshield primarily detects vehicles ahead, pedestrians, obstacles, signals, traffic signs, lanes, and the like.

[0172] By the way, Figure 33 Examples of the imaging ranges of imaging units 12101 to 12104 are shown. Imaging range 12111 represents the imaging range of imaging unit 12101, which is located on the front nose. Imaging ranges 12112 and 12113 represent the imaging ranges of imaging units 12102 and 12103, respectively, which are located on the side mirrors. Imaging range 12114 represents the imaging range of imaging unit 12104, which is located on the rear bumper or rear door. For example, by superimposing the image data captured by imaging units 12101 to 12104, a bird's-eye view image of vehicle 12100, viewed from above, can be obtained.

[0173] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0174] For example, microcomputer 12051 can determine the distance to each three-dimensional object within imaging ranges 12111 to 12114 and the temporal change in distance (relative speed with respect to vehicle 12100) based on the distance information obtained from imaging units 12101 to 12104, thereby extracting the closest three-dimensional object as the preceding vehicle. Specifically, this three-dimensional object is located on the travel path of vehicle 12100 and is traveling at a predetermined speed (e.g., equal to or greater than 0 km / h) in substantially the same direction as vehicle 12100. Furthermore, microcomputer 12051 can set a following distance to be maintained between the vehicle and the preceding vehicle, and execute automatic braking control (including follow-up stop control) or automatic acceleration control (including follow-up start control), etc. Thus, cooperative control such as automatic driving, which is intended to allow the vehicle to travel autonomously without relying on driver operation, can be executed.

[0175] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can classify 3D object data regarding 3D objects into 3D object data for two-wheeled vehicles, standard-sized vehicles, large vehicles, pedestrians, utility poles, and other 3D objects, extract the classified 3D object data, and use the extracted 3D object data to automatically avoid obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as those that the driver of the vehicle 12100 can visually identify and those that are difficult for the driver of the vehicle 12100 to visually identify. The microcomputer 12051 then determines a collision risk, representing the risk of collision with each obstacle. If the collision risk is equal to or higher than a set value, indicating a potential collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or display unit 12062, and the drive system control unit 12010 executes forced deceleration or evasive steering. The microcomputer 12051 can thus assist driving to avoid collisions.

[0176] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can identify pedestrians by determining whether a pedestrian exists in images captured by the imaging units 12101 to 12104. This identification of pedestrians can be performed, for example, by a program that extracts feature points from images captured by the imaging units 12101 to 12104, which are infrared cameras, and then performs pattern matching on a series of feature points representing the object's outline to determine whether the pedestrian is present. When the microcomputer 12051 determines that a pedestrian exists in the images captured by the imaging units 12101 to 12104 and identifies the pedestrian, the audio / video output unit 12052 controls the display unit 12062 to display a square outline superimposed on the identified pedestrian for emphasis. The audio / video output unit 12052 can also control the display unit 12062 to display an icon representing the pedestrian at a desired location.

[0177] So far, an example of a vehicle control system to which the technology according to the present disclosure can be applied has been described. For example, the technology according to the present disclosure can be applied to the imaging unit 12031 in the above-mentioned configuration. Specifically, Figure 1 The light sensing device 100 in FIG. 1 can be applied to the imaging unit 12031. Since synchronization between the output of the EVS 300 and the output of the SNN circuit 500 becomes easier by applying the technology according to the present disclosure to the imaging unit 12031, the performance of the system can be improved.

[0178] Note that the above embodiments are shown as examples for implementing the present technology, and that the matters in the embodiments and the disclosed specific matters in the claims are respectively related. Similarly, the disclosed specific matters in the claims and the matters in the embodiments of the present technology that are given the same names as the disclosed specific matters are respectively related. It should be noted that the present technology is not limited to the embodiments, but can be implemented by making various modifications to the embodiments within the scope of the present invention.

[0179] Note that the advantageous effects described in this specification are shown only as examples, but are not the only examples, and there may also be other advantageous effects.

[0180] Note that the present technology may also have the following configurations. (1) A light detection device, comprising: a sensor configured to output a plurality of first line data, wherein the first line data includes a plurality of pixel data; A neural network circuit is constructed as follows: processing at least one first row of data among the plurality of first rows of data; and outputting a second row of data including a result of the processing; and a communication interface configured to transmit a communication frame including the at least one first row of data among the plurality of first row of data and the second row of data, Wherein, the communication interface includes: a first header including identification information of at least one first row of data among the plurality of first row of data; the at least one first row of data among the plurality of first rows of data; a second header including identification information of the second row of data; and The second row of data. (2) The light detection device according to 1, wherein the first header is added to each of the at least one first line data among the plurality of first line data. (3) The light detection device according to item 1, wherein the first header is added to each of the at least one first line data group among the plurality of first line data. (4) The light detection device according to 1, wherein the second header is added to each of the second line data. (5) The light detection device according to 1, wherein the second header is added to each of the second line data groups. (6) The light detection device according to item 1, wherein the identification information includes time information. (7) The light detection device according to item 1, wherein the identification information includes row number information. (8) The light detection device according to item 6, wherein the time information includes a time at which the at least one first line data among the plurality of first line data is output. (9) The light detection device according to item 7, wherein the row information includes a row number of the at least one first row data among the plurality of first row data. (10) The light detection device according to 1, wherein the first line data among the plurality of first line data are output in an asynchronous manner. (11) The light detection device according to item 1, wherein the second line data is provided between at least one first line data among the plurality of first line data. (12) The light detection device according to item 1, wherein the sensor comprises an event-based vision sensor (EVS), a photon counting sensor, or a contact image sensor (CIS). (13) The light detection device according to item 1 further includes a first chip and a second chip. wherein the first chip is stacked on the second chip, wherein the pixels of the sensor are arranged in the first chip, and The pixel readout circuit is arranged in the second chip. (14) The light detection device according to item 13 further includes a third chip, wherein the first and second chips are stacked on the third chip, and The communication interface is provided in the third chip. (15) A processing device, comprising: The circuit is constructed as follows: receiving a communication frame from the light detection device, wherein the light detection device comprises a sensor configured to output a plurality of first line data, the first line data comprising a plurality of pixel data; and A neural network circuit is constructed as follows: processing at least one first row of data among the plurality of first rows of data; and Outputting a second row of data including the result of the processing, Wherein, the communication frame includes: a first header including identification information of at least one first row of data among the plurality of first row of data; the at least one first row of data among the plurality of first rows of data; a second header including identification information of the second row of data; and The second row of data. (16) The processing device according to 15, wherein the first header is added to each of the at least one first line data among the plurality of first line data. (17) The processing device according to item 15, wherein the first header is added to each of the at least one first line data group in the plurality of first line data. (18) The processing device according to 15, wherein the second header is added to each of the second line data. (19) The processing apparatus according to item 15, wherein the second header is added to each of the second row data groups. (20) The processing device according to item 15, wherein the identification information includes time information. (twenty one) The processing device according to item 15, wherein the identification information includes row number information. (twenty two) The processing device according to claim 20, wherein the time information includes a time when the at least one first line of data among the plurality of first line data is output. (twenty three) The processing device according to claim 21, wherein the row information includes a row number of the at least one first row data among the plurality of first row data. (twenty four) The processing device according to item 15, wherein the first line data among the plurality of first line data are output in an asynchronous manner. (25) The processing device according to item 15, wherein the second line data is set between at least one first line data among the plurality of first line data. (26) The processing device according to item 15, wherein the sensor comprises an event-based vision sensor (EVS), a photon counting sensor, or a contact image sensor (CIS). (27) The processing device according to item 25 further includes a first chip and a second chip, wherein the first chip is stacked on the second chip, wherein the pixels of the sensor are arranged in the first chip, and The pixel readout circuit is arranged in the second chip. (28) The processing device according to item 27 further includes a third chip, wherein the first and second chips are stacked on the third chip, and The communication frame is set in the third chip. (B1) A light sensing device, comprising: a sensor that outputs a predetermined number of first lines of data, each of which includes a plurality of pieces of pixel data; a neural network circuit that sequentially processes a predetermined number of the first line data based on a neural network model and outputs second line data obtained by arranging processing results of each of the predetermined number of the first line data; and A format processing section generates a communication frame in which a predetermined number of the first line data and the second line data associated with each other are stored. (B2) The light sensing device according to (B1) above further includes: a first header adding section that adds a first header including predetermined information to the first line data; and A second header adding section adds a second header including the predetermined information to the second line data. (B3) The light sensing device according to the above (B2), wherein the first header adding section adds the first header to each of a predetermined number of the first line data. (B4) The light sensing device according to the above (B2), wherein the first header adding section adds the first header to the first line data at the beginning of a predetermined number of the first line data. (B5) The light sensing device according to (B2) or (B3) above, wherein the second header adding section adds the second header to each of a predetermined number of the second line data. (B6) The light sensing device according to (B2) or (B3) above, wherein the second header adding section adds the second header to the second line data at the beginning of a predetermined number of the second line data. (B7) The light sensing device according to any one of (B2) to (B6) above, wherein the predetermined information includes time information indicating a time when the first row of data is output. (B8) The light sensing device according to any one of (B2) to (B6) above, wherein The pixel data includes time information, and The predetermined information includes identification information about a row corresponding to the first row data. (B9) The light sensing device according to any one of the above (B8), further comprising: A digital processing section performs predetermined processing on the first line data to which the first header is added and the second line data to which the second header is added, and supplies the first line data and the second line data to the format processing section. (B10) The light sensing device according to any one of (B2) to (B8) above, comprising: a first digital processing section that performs predetermined processing on the first line data to which the first header is added, and supplies the first line data to the format processing section; and A second digital processing section performs predetermined processing on the second line data to which the second header is added, and supplies the second line data to the format processing section. (B11) The light sensing device according to any one of (B2) to (B10) above, further comprising: A FIFO (First In First Out) memory holds the first line data to which the first header is added and the second line data to which the second header is added by a first in first out scheme. (B12) The light sensing device according to any one of (B2) to (B10) above, further comprising: a first FIFO memory that holds the first line data to which the first header is added by a first-in-first-out scheme; and A second FIFO memory holds the second line data to which the second header is added through a first-in-first-out scheme. (B13) The light sensing device according to the above (B12), wherein the neural network circuit reads the first row of data from the first FIFO memory. (B14) The light sensing device according to the above (B12), wherein the sensor outputs the first row data to the first FIFO memory and the neural network circuit. (B15) The light sensing device according to any one of (B1) to (B14) above, wherein the sensor includes an EVS (Event-based Vision Sensor). (B16) The light sensing device according to any one of (B1) to (B14) above, wherein the sensor includes a photon measurement circuit that counts photons. (B17) The light sensing device according to any one of (B1) to (B14) above, wherein the sensor includes a CIS (CMOS Image Sensor). (B18) The light sensing device according to any one of (B1) to (B17) above, wherein the sensor, the neural network circuit and the format processing unit are arranged to be distributed on a plurality of stacked chips. (B19) A light sensing device, comprising: a sensor that outputs a predetermined number of first lines of data, each of which includes a plurality of pieces of pixel data; a neural network circuit that sequentially processes a predetermined number of the first line data based on a neural network model and outputs second line data obtained by arranging processing results of each of the predetermined number of the first line data; a header adding section that adds a header including predetermined information about the first line data to the second line data; and A format processing section generates a communication frame in which the second line data to which the header is added is stored. (B20) A method for controlling a light sensing device, comprising: The sensor outputs a predetermined number of first lines of data, each of which includes a plurality of pixel data; A process of sequentially processing a predetermined number of the first line data based on a neural network model by a neural network circuit, and outputting second line data obtained by arranging the processing results of each of the predetermined number of the first line data; and A process of generating, by a format processing section, a communication frame in which a predetermined number of the first line data and the second line data associated with each other are stored. Reference Signs List

[0181] 100: Light sensing device 110: Optical Department 120: Digital Signal Processor (DSP) Circuit 130: Display unit 140: Operation Department 150: Bus 160: Frame Memory 170: Storage 180: Power Supply 191: System global clock supply unit 192: External Sensor 200: Sensor chip 201: Pixel chip 202, 203: Circuit Chip 211, 212: Header addition 213: Register 221, 222, 541 to 545: First-in, first-out (FIFO) memory 230, 240: Test pattern generation unit 231: Test pattern supply unit 232: Switch 251, 252: Digital Processing Unit 260: Format Processing Department 261: Buffer memory 262: Rearrangement Processing Department 263: Formatter 270: External communication interface 300: Event-based Vision Sensor (EVS) 310, 610: Drive unit 320, 620, 740: Pixel array 330, 640, 720: Timing control circuit 340: Line Scanner 400, 630, 750: pixels 410: Pixel circuit 411, 751: Photodiode 412, 413, 435, 442, 444: Negative-channel metal-oxide-semiconductor (nMOS) transistors 414, 421, 422, 432, 434, 441, 443: Positive channel (pMOS) transistors 420: Buffer 430: Differential Circuits 431, 433: Capacitors 440: Quantizer 500: Simulated Neural Network (SNN) Circuit 511: Input layer 512: Middle layer 513: Output layer 520: Input / Output (I / O) Interface 530: Multi-core Array 540: Router 546: Arbitrator 550: Core 551: Core router 552: Neuron I / O 553: Product-Sum Unit 554: Working memory 555: Membrane Potential Memory 556: Leaky Integrator Trigger (LIF) Unit 600: Photon Measurement Circuit 631: Quenching resistor 632: Single Photon Avalanche Diode (SPAD) 633: Inverter 634: Photon Counter 650: Reading processing unit 700: Contact Image Sensor (CIS) 710: Vertical scanning circuit 730: Digital-to-Analog Converter (DAC) 752: Pass transistor 753: Reset transistor 754: Floating Diffusion Layer 755: Amplifier transistor 756: Select transistor 760: Column Analog-to-Digital Converter (ADC) 770: Horizontal transfer scanning circuit

Claims

1. A light detection device, comprising: a sensor configured to output a plurality of first line data, wherein the first line data includes a plurality of pixel data; A neural network circuit is constructed as follows: processing at least one first row of data among the plurality of first rows of data; and outputting a second row of data including a result of the processing; as well as a communication interface configured to transmit a communication frame including the at least one first row of data among the plurality of first row of data and the second row of data, Wherein, the communication interface includes: a first header including identification information of at least one first row of data among the plurality of first row of data; the at least one first row of data among the plurality of first rows of data; a second header including identification information of the second row of data; and The second row of data. 2 . The light detection device according to claim 1 , wherein the first header is added to each of the at least one first line data among the plurality of first line data. 3 . The light detection device according to claim 1 , wherein the first header is added to each of the at least one first line data group among the plurality of first line data. The light detection device according to claim 1 , wherein the second header is added to each of the second line data. 5 . The light detection device according to claim 1 , wherein the second header is added to each of the second line data groups. The light detection device according to claim 1 , wherein the identification information includes time information. The light detecting device according to claim 1 , wherein the identification information includes row number information.

8. The light detection device according to claim 6, wherein The time information includes a time at which the at least one first line of data among the plurality of first line data is output. 9 . The light detection device according to claim 7 , wherein the row information includes a row number of the at least one first row data among the plurality of first row data.

10. The light detection device according to claim 1, wherein The first line data among the plurality of first line data are output in an asynchronous manner.

11. The light detection device according to claim 1, wherein The second line data is provided between at least one first line data among the plurality of first line data. 12 . The light detection device according to claim 1 , wherein the sensor comprises an event-based vision sensor (EVS), a photon counting sensor, or a contact image sensor (CIS).

13. The light detection device according to claim 1, further comprising a first chip and a second chip, wherein the first chip is stacked on the second chip, wherein the pixels of the sensor are arranged in the first chip, and The pixel readout circuit is arranged in the second chip.

14. The light detection device according to claim 13, further comprising a third chip, wherein the first and second chips are stacked on the third chip, and The communication interface is provided in the third chip.

15. A processing device comprising: The circuit is constructed as follows: receiving a communication frame from the light detection device, wherein the light detection device comprises a sensor, the sensor being configured to output a plurality of first line data, the first line data comprising a plurality of pixel data; and A neural network circuit is constructed as follows: processing at least one first row of data among the plurality of first rows of data; and Outputting a second row of data including the result of the processing, Wherein, the communication frame includes: a first header including identification information of at least one first row of data among the plurality of first row of data; the at least one first row of data among the plurality of first rows of data; a second header including identification information of the second row of data; and The second row of data. 16 . The processing device according to claim 15 , wherein the first header is added to each of the at least one first line data among the plurality of first line data. 17 . The processing device according to claim 15 , wherein the first header is added to each of the at least one first line data group in the plurality of first line data. The processing device according to claim 15 , wherein the second header is added to each of the second line data.

19. The processing apparatus according to claim 15, wherein the second header is added to each of the second row data groups.

20. The processing device according to claim 15, wherein the identification information includes time information. The processing device according to claim 15 , wherein the identification information includes row number information.

22. The processing device according to claim 20, wherein The time information includes a time at which the at least one first line of data among the plurality of first line data is output.

23. The processing device according to claim 21, wherein The row information includes a row number of the at least one first row of data among the plurality of first row of data.

24. The processing device according to claim 15, wherein The first line data among the plurality of first line data are output in an asynchronous manner.

25. The processing device according to claim 15, wherein The second line data is provided between at least one first line data among the plurality of first line data.

26. The processing device of claim 15, wherein the sensor comprises an event-based vision sensor (EVS), a photon counting sensor, or a contact image sensor (CIS).

27. The processing device according to claim 25, further comprising a first chip and a second chip, wherein the first chip is stacked on the second chip, wherein the pixels of the sensor are arranged in the first chip, and The pixel readout circuit is arranged in the second chip.

28. The processing device according to claim 27, further comprising a third chip, wherein the first and second chips are stacked on the third chip, and The communication frame is set in the third chip.

Images