Image sensor using an array of single-photon avalanche diode photodetectors.

Abstract

The present invention provides a pixel array layer including a plurality of pixel segments having a plurality of pixels for detecting photons, each pixel segment providing a digital pixel output; a processing layer including a number of processing cores, each associated with one of a plurality of pixel segments for receiving pixel outputs for pixels of the respective pixel segment, each processing core in bidirectional communication with one or more adjacent processing cores; The processing cores relate to a stacked arrangement of an image sensor device, each configured to receive pixel outputs of pixels of an associated pixel segment and to distribute processing of the pixel outputs between the processing core and at least one adjacent processing core as an adjacent processing core.

Description

[Technical Field]

[0001] The present invention relates to image sensors, and more particularly to image sensors that use photon detection with single photon avalanche diodes and have improved configurable image data processing flexibility. [Background technology]

[0002] Generally, an image sensor includes a two-dimensional array of photodetectors. The photodetectors can be configured to detect one or more impinging photons and provide corresponding photon detection signals to generate image information of the received light distribution. One type of photodetector includes a single-photon avalanche diode (SPAD), which is configured to detect light upon photon detection by generating electron-hole pairs and multiplying them through an electric field to create a detectable avalanche of electrons.

[0003] Such so-called Geiger-mode photodetector cells are typically fabricated on / in a silicon substrate with a pn junction electrically biased beyond its breakdown voltage so that each electron-hole pair can trigger an avalanche multiplication process that forms the photon detection signal as an electrical pulse signal.

[0004] After recognizing such an avalanche, it is quenched by reducing the electric field that accelerated the generated electrons so that the avalanche process stops, and then the electric field is increased again, making the photodetector cell ready for the next photon detection.

[0005] U.S. Patent Publication No. 9,210,350 discloses an imaging system including a pixel array including a plurality of pixels, each of which includes a single photon avalanche diode (SPAD) coupled to detect photons in response to incident light. A readout circuit includes a plurality of photon counters coupled to each of the plurality of pixels to count the number of photons detected by the respective plurality of pixels. Each of the plurality of photon counters is coupled to stop counting photons for each of the plurality of pixels that has reached a threshold photon count, and each of the plurality of photon counters is coupled to continue counting photons for each of the plurality of pixels that has not reached the threshold photon count. A control circuit is coupled to the pixel array to control operation of the pixel array, and includes an exposure time counter coupled to count the exposure time elapsed before each of the plurality of pixels detects the threshold photon count. Respective exposure time counts and photon counts are combined for each of the plurality of pixels in the pixel array.

[0006] A C Ulku, C. Bruschini, I. M Antolovic et al., "A 512 × 512 SPAD image sensor with integrated gating for widefield FLIM," IEEE Journal of Selected Topics in Quantum Electronics, vol. 25, no. 1, pp. 1-12, 2019, discloses an image sensor with 512 × 512 photon-counting pixels, each containing a single-photon avalanche diode (SPAD), a 1-bit memory, and a gating mechanism that can turn the SPAD on and off. The sensor is designed to achieve high frame rates.

[0007] K. Morimoto, A. Ardelean, M.-L. Wu, et al., “Megapixel time-gated SPAD image sensor for 2D and 3D imaging applications”, Optica, vol. 7, no. 4, pp. 346-354, 2020, discloses a 1 Mpixel single-photon avalanche diode camera featuring a 3.8 ns time gate and a 24 kfps frame rate.

[0008] C. Zhang, S. Lindner, I. M Antolovic, M. Wolf, and E. Charbon, “A CMOS SPAD imager with collision detection and 128 dynamically reallocating TDCs for single-photon counting and 3D time-of-flight imaging,” Sensors, vol. 18, no. 11, 2018, discloses a single-photon avalanche diode (SPAD) sensor with a per-pixel time-to-digital converter (TDC) structure to achieve high photon throughput. A 32 x 32 pixel SPAD sensor fabricated in 180 nm CMOS image sensor technology is disclosed, and a dynamically reallocated TDC is implemented to achieve the same photon throughput as a per-pixel TDC. Each of the four TDCs is shared by 32 pixels via a collision detection bus.

[0009] S. Lindner, S. Pellegrini, Y. Henrion, B. Rae, M. Wolf, and E. Charbon, "A high-PDE, backside-illuminated SPAD in 65 / 40-nm 3D IC CMOS pixel with cascaded passive quenching and active recharge," IEEE Electron Device Letters, vol. 38, no. 11, pp. 1547-1550, 2017, discloses a detector pixel based on a single-photon avalanche diode (SPAD) fabricated in backside-illuminated (BSI) three-dimensional IC technology. The chip stack includes an image detection layer fabricated in 65-nm image sensor technology and a data processing layer fabricated in 40-nm CMOS. Using simple CMOS-compatible technology, the pixel can perform passive quenching and active recharging at voltages far exceeding those imposed by a single transistor, while ensuring that reliability limits across gate-source (VGS), gate-drain (VGD), and drain-source (VDS) are not exceeded for any device. Summary of the Invention [Problem to be solved by the invention]

[0010] It is an object of the present invention to provide an improved structure for an image sensor device having a stack with an image detection layer and a data processing layer that provides widely configurable modes of operation for efficient processing.

[0011] This object is achieved by an image sensor arrangement according to claim 1. Further embodiments are set forth in the dependent dependent claims. [Means for solving the problem]

[0012] According to a first aspect, an image sensor device having a stacked arrangement includes: a pixel array layer including a plurality of pixel segments each having a plurality of pixels for photon detection providing a digital pixel output; a processing layer including a number of processing cores each associated with one of a plurality of pixel segments for receiving pixel outputs of pixels of the respective pixel segment, the processing cores each in bidirectional communication with one or more adjacent processing cores; Each of the processing cores is configured to receive pixel outputs for pixels of an associated pixel segment, and to distribute processing of the pixel outputs between the processing core and at least one of the adjacent processing cores.

[0013] The image sensor device is a reconfigurable and scalable computational image sensor with fully autonomous processing capabilities provided by the processing cores. The flexibility of the architecture comes from the ability to run custom programs / algorithms in each processing core, but also from reconfigurable hardware at the pixel interface that can be customized via software at run time.

[0014] Furthermore, the pixels of the pixel array layer may include a detector diode, in particular a SPAD, and a pre-processing circuit coupled to the detector diode to provide a respective pixel output as a signal, in particular the signal may be provided to the processing layer via the pixel array layer, for example, the pixel array layer is provided on a semiconductor (e.g. silicon) substrate that is processed by semiconductor processing techniques to produce the pixel structure and the pre-processing circuit.

[0015] The processing layer may also be provided on a semiconductor (eg silicon) substrate that is processed by semiconductor processing techniques to create the pixel structure and the pre-processing circuitry.

[0016] According to one embodiment, each processing core may include a front-end block that may include freely configurable combinatorial logic and / or at least one look-up table to provide masking and / or logical operations on the pixel outputs for pre-processing and / or combining the pixel outputs.

[0017] A timing block may be configured to receive the pre-processed pixel output and perform various timing functions known in the art, such as pulse width and / or phase shift measurements.

[0018] Further, the processing block in each of the processing cores may include a set of general-purpose registers, an arithmetic and logic unit (ALU), a RAM, and a control block for providing flexible data processing capabilities, the control block being configured to control the data processing operations of the processing block.

[0019] In particular, the control block of at least one processing core associated with a pixel segment may be configured to divide one or more processing tasks to be performed on pixels of that pixel segment into processing portions in which at least two processing portions can be processed simultaneously in parallel, and the at least two parallel-processable processing portions are executed in the processing core and at least one adjacent processing core by directly controlling the processing of each processing portion in the processing block of the processing core and instructing at least one adjacent processing core to respectively execute the processing of the other of each processing portion.

[0020] According to one embodiment, the processing tasks may be LSTM calculations performed on images detected by an image sensor device, the LSTM calculations including matrix operations, addition operations, sigmoid operations and tangent hyperbolic operations, and the control block of each processing core respectively: - dividing the execution of the matrix and summation operations into several processing parts to be executed on the pixels of the one associated pixel segment; - executing at least one of a plurality of processing portions in each control block; - may be configured to communicate at least one of the plurality of processing portions to at least one of the adjacent processing cores, and to instruct the at least one adjacent processing core to execute a respective at least one of the plurality of processing portions.

[0021] The embodiments will be explained in more detail with reference to the accompanying drawings. [Brief explanation of the drawings]

[0022] [Figure 1] 1 shows a top view of an image sensor. [Figure 2] 1 shows a cross-sectional view of an image sensor. [Figure 3] FIG. 2 shows a circuit diagram of a pixel circuit in a pixel array layer. [Figure 4] 1 shows a schematic diagram of a pixel layout. [Figure 5] 1 shows a detailed top view of the edge of the processing layer substrate. [Figure 6] 1 shows a block diagram of a processing core. [Figure 7] 1 shows a schematic diagram of a reconfigurable front-end circuit block. [Figure 8] 10 shows the group arrangement of pixel outputs associated with the LUT. [Figure 9] 1 shows a schematic diagram of the timing blocks implemented in each processing core. [Figure 10] The processing blocks implemented in each processing core are shown. [Figure 11] 1 shows the structure of an image sensor including multiple processing cores arranged in a grid. [Figure 12] 1 shows a scheduled graph of equations that implement the LSTM inference operation. [Figure 13] 1 shows an exemplary arrangement of five processing cores with one master and four slave processing cores. DETAILED DESCRIPTION OF THE INVENTION

[0023] Figure 1 shows a top view of an image sensor device 1, and Figure 2 shows a cross-sectional view of the image sensor device 1 with a stack of a pixel array layer 2 and a processing layer 3. Both layers 2, 3 can be manufactured for example in CMOS technology and / or FinFET 3D technology on a silicon substrate.

[0024] The pixel array layer 2 may be an exemplary 12x24 array (or any other size) of SPAD pixels 21 (SPAD: Single Photon Avalance Diodes) grouped into pixel segments 22 of 4x4 SPAD pixels. Each pixel output is electrically connected to the processing layer 3 using through-substrate vias (TSVs) 23.

[0025] Processing layer 3 has a 3x6 array of independent processing cores 31, each connected to a SPAD pixel 21 in a respective pixel segment 22. Processing cores 31 can share information and exchange data with their directly neighboring processing cores 31 and can synchronize with each other through the use of internal and external handshaking signals. Processing layer 3 is designed for 3D integration with large TSV landing sites 32 that are similar in structure to conventional flip-chip ball bonding pads.

[0026] The processing layer 3 contains digital processing electronics and serves as a front end to process the pixel array raw data (including the pixel output of each SPAD pixel 21) of the pixel array layer 2. Such a stacked architecture offers the possibility to use the processing layer 3 as a general-purpose readout IC coupled with custom detector technologies that are not limited to SPAD-based pixel arrays.

[0027] 3 shows an exemplary pixel circuit diagram as the electronic circuit of the pixel array layer 2. A detector diode D1, such as a SPAD, is connected to a high voltage potential V HVand ground potential GND, in series with cascode transistor T1 and reset transistor T2. Cascode transistor T1 is used to extend the bias voltage range of the pixel, and reset transistor T2 performs a clocked active recharge controlled by the applied RST signal.

[0028] When an avalanche occurs, the voltage at node A between transistors T1 and T2 rises and, depending on the state of gate transistor T3 coupled to node A as a transmission gate, can affect node B and the gate of transistor T5. When node B is high, transistor T5 drives the gate of oversized (thick oxide) transistor T8, which discharges the large parasitic capacitance of TSV23. The RST signal also drives transistor T4 coupled to node B to reset node B and transistor T6 coupled in series with transistor T4 and transistor T7 coupled in series with transistor T8 to recharge the capacitance of TSV23. Voltage level translation is achieved, for example, by setting the supply voltage VDDBOT to 0.8V plus the threshold voltage of transistor T8.

[0029] Figure 4 shows a schematic pixel layout. The detector diode 21a is located in the center of the pixel area. The octagonal shape at the bottom left is the metal contact for the TSV 23. All pixel circuitry is located in the rectangular area 21b at the bottom and left. Due to the small area and spacing requirements, all transistors can be formed as thick-oxide NMOS to avoid minimum spacing rules between different types of transistors. The pixel areas of adjacent pixels are indicated by dotted lines.

[0030] Processing layer 3 can accommodate TSV landing sites 32, which can be formed by a group of 16 rectangular aluminum contacts through a set of ESD protection diodes (not shown) to the underlying processing core 31. TSV landing sites 32 receive the pixel output.

[0031] 5 shows a detailed top view of the edge of the handle layer substrate, where TSV landing sites 32 can be seen next to regular-sized bond pads 33 for external connection of the image sensor 1.

[0032] 6 shows a schematic block diagram of the processing cores 31. Each processing core 31 is connected to 16 pixels on the pixel array layer 2 arranged in a 4x4 pixel pattern. The pixel output signals (pixel outputs) coming from the pixel circuits on the pixel array layer 2 are connected to a reconfigurable front-end block 41 which includes combinational logic and a look-up table (LUT).

[0033] Front-end circuitry block 41 can pre-process the pixel output data before it is transferred to timing block 42 and / or processing block 43. Timing block 42 is specialized circuitry that can perform various timing functions such as pulse width or phase shift measurements.

[0034] Processing block 43 is a set of general purpose registers 431, an arithmetic logic unit (ALU) 432 and RAM 433. The overall operation of processing core 31 is coordinated by control block 44, which includes instruction ROM 441 and instruction decoder 442 that define the operation of processing block 43.

[0035] The control block 44 may receive inputs from adjacent processing cores 31 and may then provide software control outputs used for synchronization. Similarly, the timing block 42 may receive inputs from adjacent processing cores 31 and may provide high speed propagation signals via other processing cores 31, for example, via dedicated channels external to the device.

[0036] Synchronization works for the front-end block 41 and timing block 42 by implementing dedicated signal paths (1-bit wires) that can route the output of the front-end or timing module to the front-end and / or timing block. Thus, the timing blocks 42 of all adjacent processing cores 31 can be triggered by the detection of the same photon by the master. In this case, the output of the front-end circuit block 41 of the master processing core 31 is routed via a multiplexer to all four adjacent processing cores 31 and can be treated as if it came from its own front-end circuit block 41 (of the slave processing core 31).

[0037] If a range greater than 16 bits is required for timing block 42, the most significant bit of timing block 42 is routed to timing block 42 of an adjacent processing core 31 where it is used as a counter clock, effectively using two separate 16-bit counters as a single 32-bit counter.

[0038] The synchronization signal is generated by the control block 44 as a simple 1-bit signal coming from the neighboring processing core 31 that can be checked using a conditional instruction in the code, e.g., if the neighboring processing core 31 sends a signal, jump to row XX. There are also dedicated instructions to send a signal to a specific neighboring processing core 31, e.g., a strobe signal to the neighboring processing core 31.

[0039] Figure 7 shows a schematic diagram of the reconfigurable front-end circuit block 41. The 16 pixel outputs PXL[0...15] are connected to four groups of four LUTs 411 (LUT0 to LUT3) according to the diagram shown in Figure 8. The purpose is to create the possibility of binning 4x4 pixels into 2x2 groups. Each LUT can be programmed by the control block 44 using four instruction cycles to implement any logical function of the following type:

[0040] Q3Q2Q1Q0=a[0]P3P2P1P0+a[1]P3P2P1P0+...+a

[15] P3P2P1P0

[0041] where P and Q are the 4-bit LUT input and output respectively, and a[] is an array of 16 values ​​of 0 and 1.

[0042] The outputs of the first layer of LUTs, LUT0, LUT1, LUT2, and LUT3, are connected to the second layer of circuitry, which includes adder 412 and LUT4 (413). The adder combines the 16 bits of LUT411's output Q into a single 5-bit number, effectively counting the number of seconds. LUT4 is a larger version of the other 4, with an 8-bit input and a 1-bit output. In contrast to the adder, only the most significant 2 bits from the output of the previous LUT 411 are connected.

[0043] The LUT 4 can also be configured by the control block 44 in 16 instruction cycles to implement any of the following types of functions:

[0044] Q0=a[0]P7P6P5P4P3P2P1P0 +a[1]P7P6P5P4P3P2P1P0 +... +a

[0255] P7P6P5P4P3P2P1P0 where P is an 8-bit input assembled by the 2 bits of the output of LUT 411, Q is a 1-bit output, and a is an array of 256 values ​​of 0 and 1.

[0045] The 16 pixel outputs are connected to two OR trees 414 and can be individually masked using a set of AND gates 415. This secondary path is designed with an independent set of constraints to allow high speed signal propagation and can serve as an input to timing block 42 or other adjacent processing cores 31.

[0046] Various signals from the front-end circuit block 41, such as, but not limited to, the LUT and adder outputs and raw pixel values, as well as all other circuitry, are connected to an output DMUX 416 which can be controlled by the control block 44. This allows software flexibility to select various pre-processed versions of the inputs without having to reconfigure the front-end circuit block 41.

[0047] 9 shows a schematic diagram of the timing block 42 implemented in each processing core 31. A 16-bit counter 421 serves as the central element of the timing block 42, and its value is used as the timing block output. A set of multiplexers 422 is used to select the sources that serve as the counter clock signal C and the enable signal EN, with a wide range of choices available in both cases. The inputs to the multiplexers 422 may be the high-speed output signals of FIG. 7. The counter clock signal and the enable signal may be selected from the front-end block 41 in a variety of ways.

[0048] The enable signal EN may be supplied directly from the front-end circuit block output or via an SR latch 423 that can combine two separate input signals at the front-end output. Additionally, pulses generated by the adjacent processing core 31 or the control block 44 can be used as the counter clock signal C or the enable signal EN.

[0049] The local oscillator 424 can be formed by a ring of many (eg, seven) NAND gates and can be used to generate a higher frequency clock reference for the counter.

[0050] Timing block 42 generates two flags that can be used by control block 44 for conditional instructions: Counter Overflow CO and Latch Set LS. Counter Overflow CO is set when an overflow is detected in counter 421 and is essentially a latched 17th counter bit. Latch Set LS is the state of the Input SR latch and can be used to detect the arrival of an input.

[0051] The control block 44 can reconfigure the functionality of the timing block 42 by setting all multiplexers 422 and resetting the counter 421 and the two flags CO and LS. Complete reconfiguration requires two instruction cycles, but in most cases a single cycle is sufficient.

[0052] 10, processing block 43 includes a number of general purpose registers 431, a byte selector block 432, an ALU 433, and a RAM 434. The input of processing block 43 connects to front end block 41, timing block 42, and adjacent processing cores via a set of multiplexers managed by control block 44. The output is a RAM memory itself that can be read from processing core 31 by external system circuitry or a set of registers that connect to adjacent processing core 31.

[0053] General purpose registers 431 can be loaded with data coming from the input, ALU 433, or RAM 434. The load signals of general purpose registers 431 are driven independently by control block 44 to allow the same data to be written to multiple locations simultaneously if desired.

[0054] The byte selector block 432 is a specialized circuit used to shift or extract specific bytes from the data word presented at its input. It can be used alone with specialized instructions or in combination with other operations. The following table summarizes the operations that the byte selector block 432 can perform:

[0055] Function # Output effect 1 I[31:0] No operation 2 I[7:0] Byte 0 3 I[15:8] Byte 1 4 I[23:16] Byte 2 5 I[31:24] Byte 3 6 I[15:0] Lower half 7 I[31:16] Upper half 8 I[0:31] Reverse bit order

[0056] The ALU 433 is a combinational circuit block with three equal-bit-sized inputs and one output. The ALU control signal selects the ALU, which uses one of 25 possible operations to compute its output. The following operations can be performed: NOT O = NOT Ain, AND O = Ain AND Bin, OR O = Ain OR Bin, XOR O = Ain XOR Bin, NEG O = -Cin, ADD O = Ain + Cin, SUB O = Ain - Cin, MUL O = Ain × Bin, MAC O = Ain × Bin + Cin, CMP Ain <Bin、RL O=Ain[30:0]&Ain

[31] 、RR O=Ain[0]&Ain[31:1]、SL O=Ain<<1、SR O=Ain> >1, MAX O=max(Ain,Bin), MIN O=min(Ain,Bin).

[0057] In addition to the integer output, ALU 433 also generates two flags, 0 and Carry, that are used for conditional jumps or instruction calls. Depending on the result of an arithmetic operation, these flags are set or cleared and remain the same until another operation acts on them. As an exception, none of the logical operations affect the Carry flag.

[0058] The inputs Ain, Bin, and Cin to the ALU can come from multiple sources: general-purpose registers 432, explicit RAM addresses, pointers to RAM addresses, or hard-coded values ​​in the instruction code. Similarly, the result of an operation can be written to a register or to an explicit or specified RAM location.

[0059] The memory may be a dual-port RAM block. The read and write address ports are independently controlled to allow for the aforementioned flexibility of data sources. Additionally, the RAM can be accessed externally by overriding all connections, a feature used for debugging and extracting processor output.

[0060] The control block 44 includes an instruction memory 441 formed as a ROM, an instruction decoder circuit 442, and a finite state machine 443. A 1-bit synchronization signal from each adjacent processing core 31 can be used during the execution of conditional instructions. Similarly, a 1-bit output signal is connected to each adjacent processing core 31 and can be strobed or set via software. In addition, there are two 1-bit inputs and a 1-bit output that can operate in the same manner as the connections to adjacent processing cores 31 and are designed to synchronize with modules external to the device.

[0061] Instruction memory 441 may be a 256 x 24 bit dual-ported RAM block. In contrast to the scratchpad RAM from processing block 43, only the read port of instruction memory 441 is accessible by processing core 31, and as a result, it behaves like a ROM. The write port may be connected to the device bus and is only used during setup or in special cases where program execution is suspended and instruction memory 441 is rewritten at run time.

[0062] The processing core 31 can be configured to follow a fetch-decode-execute sequence that takes exactly three clock cycles for each instruction. During the fetch phase, the instruction pointed to by the program counter register (PC) is read from ROM 441 and passed to the instruction decoder 442. The instruction decoder is a combinational circuit that drives all the processing core control signals. During the decode phase, in addition to setting the control signals, the required data is fetched from RAM by either driving the RAM address bus directly or by using a general-purpose register as a pointer. At the end of the final phase, the result of the operation is written to the required destination and the PC is incremented.

[0063] Each instruction may be 24 bits long and begins with a variable-length opcode, followed by a payload.

[0064] The image sensor structure includes a number of processing cores 31 arranged in a grid, such as a 6x3 grid in this embodiment, as shown in Figure 11. Each processing core 31 is connected to its four direct neighboring processing cores 31 via a bidirectional data bus 33 and a bidirectional timing signal line 34. If one or more neighbors are missing (in the case of edge cores), the corresponding signals may be connected to registers 35.

[0065] Programming and reading of the device can be performed over a conventional AXI bus, with the output of each processing core 31 being mapped to a memory location which may include instruction ROM, RAM and associated registers as required.

[0066] The two input signals are distributed to all processing cores 31 and can reach each processing core 31 in parallel and can be used for synchronization purposes.

[0067] Programming of each processing core 31 may be performed by uploading the program to a separate instruction memory (ROM) 441 via the AXI bus. To avoid unexpected behavior, the target processing core 31 should be held in a reset state during this procedure. However, two exceptions may be provided when a processing core 31 can be reprogrammed at run time: when it is certain that a portion of the program will not execute until reprogramming is finished, or when a processing core 31 is kept in a frozen state waiting for an external stimulus using a special-purpose WAIT instruction.

[0068] Instruction types can be divided into five categories: logical, arithmetic, operational, flow, and special. Most instructions have multiple variants depending on the source of the operands and the destination of the result.

[0069] Logic instructions can have one or two operands, which are sourced from registers or RAM locations. The result can be written to any register or RAM location, including the one that served as the source. No explicit operands are supported for this type of operation. After execution, the ALU carry flag is cleared regardless of the result. These instructions replace the dedicated clear flag command. The ALU zero flag functions as normal. The four logical operations supported by the architecture are NOT, AND, OR, and XOR.

[0070] The ALU can perform the following arithmetic instructions: negate, add, subtract, multiply, MAC, MAX, MIN, and compare values. Like the logical instructions, they can operate on data from general-purpose registers or RAM, but can also use three operands (MAC instruction) or explicit values ​​(ADD and SUB instructions).

[0071] Manipulation instructions operate on a single operand and are used to apply rotates and shifts or to select specific bytes. This category also includes RAM STORE and FETCH instructions, which transfer data from general-purpose registers 432 to RAM or vice versa, the latter option supporting multiple destinations simultaneously. Additionally, a LOAD instruction is provided to write an explicit value to any of the general-purpose registers 432.

[0072] Flow instructions affect program execution by modifying the PC register. The JUMP instruction can be used to jump to any address in instruction memory, either unconditionally or depending on the state of available flags. The CALL and RET instructions can be used to execute subroutines. The former operates exactly like the JUMP instruction and its variants, but pushes the PC value onto the stack so that program execution can resume from the same point when RET is called.

[0073] The highly customized architecture requires a special instruction set not typically encountered in other CPUs. Communication with adjacent processing cores 31 and external circuitry can be achieved via the SAVEN, GETN, PUTN, and TELL instructions. The first two are used to sample adjacent data buses and transfer values ​​into general-purpose registers 432. Both instructions support multiple sources and destinations simultaneously. The PUTN instruction latches the value of a general-purpose register into one or more adjacent data buses. Finally, TELL is used to strobe or set synchronization signals for adjacent control units or external IO pads.

[0074] Data from the timing module or front end can be read using dedicated GETC and GETP instructions which support simultaneous byte selection and multiple destinations. Every combinational logic path has its own configuration instructions, starting from the front end multiplexer (SETFM, SETTM) and LUT function (SETLUT) and ending in the fast path OR tree (SETOR) and timing module (SETTIME).

[0075] Finally, a special WAIT instruction can be provided to facilitate simultaneous synchronization of cores with asynchronous external trigger conditions. Execution of this instruction freezes the control block 44 in the execution stage until the specified condition is met, after which operation resumes immediately on the next clock cycle. The condition is verified by monitoring adjacent processing cores 31 and external synchronization signals, and is met only if pulses are detected from all of the requested sources, regardless of order.

[0076] A possible application is described for LSTM lidar sensors. Long short-term memory (LSTM) is a special type of artificial neural network containing feedback connections that allows for the processing of data sequences such as audio or video signals. Recently, research has focused on extending the use of LSTM to LiDAR applications, where data streams generated by time-of-flight (ToF) image sensors are processed by such networks to determine a depth map of the detected scene. The unique properties of the above architecture enable the implementation of LSTM because the processing cores 31 can share information among themselves, thereby enabling a high degree of parallelization, and the reconfigurable front-end block 41 can implement preprocessing techniques such as coincidence detection without speed or processing penalties.

[0077] As an example, an image sensor is proposed that functions as a single-point ToF detector used in an XY scanning setup and implements LSTM cells of size 20. The front-end block 41 is configured to trigger the timing block 42 with the first detected input pulse within the exposure window (trigger when any of the 4x4 pixel segments, or only one pixel segment, or at least some pixels from the 4x4 pixel segment are activated).

[0078] The following equation describes the LSTM at time step t: JPEG2025535132000002.jpg4379 where, JPEG2025535132000003.jpg12150 and JPEG2025535132000004.jpg6150 is a 20x1 array of forget, input, and output gate values, JPEG2025535132000005.jpg6150 and JPEG2025535132000006.jpg6150 is a 20x1 vector representing the hidden and cell states that are saved from one LSTM iteration to the next, JPEG2025535132000007.jpg6150 is the input value given by the timing block, JPEG2025535132000008.jpg12150 is a 20x21 weight matrix, JPEG2025535132000009.jpg12150 is a 20x1 bias array. JPEG2025535132000010.jpg6150 and The values ​​of JPEG2025535132000011.jpg6150 are all constants determined before execution during LSTM training. σ and tanh are the sigmoid and hyperbolic tangent functions, while ||, ×, and . represent function concatenation, matrix multiplication, and Hadamard multiplication, respectively.

[0079] Figure 12 shows a graphical representation of the above equations. Step S0 is simple because the concatenation operation can be replaced with a memory write. Steps S1 and S2 are the most resource-intensive, requiring a total of 420 fixed-point MACs. Steps S4, S5, and S7 require only 40 fixed-point multiplications and 20 fixed-point additions / multiplications, respectively. The nonlinear tanh and σ functions can be implemented as LUTs, i.e., simple memory read operations. To improve execution speed, steps S1 and S2 are distributed across four separate slave cores, while the remaining steps are assigned to a single master core.

[0080] The total RAM available to each core was 512 bytes, and as a result, all weight and bias coefficients had to be stored as 8-bit signed fixed-point numbers with 3 fractional bits, four coefficients per RAM word. JPEG2025535132000012.jpg12150 and JPEG2025535132000013.jpg6150JPEG2025535132000014.jpg6150 and The values ​​of JPEG2025535132000015.jpg6150 have a 16-bit signed fixed-point representation with 3 fractional bits and are stored in pairs at each RAM location. The LUT used for the nonlinear activation function has the same data format as the previous variables.

[0081] Figure 13 shows the arrangement of five processing cores 31 used to implement the LSTM. The master processing core 31 is surrounded by slave processing cores 31 on all four cardinal directions to enable the fastest possible data transfer. When the master processing core 31 finishes its exposure period, it transfers the x[t] variable to the four slaves and waits until all matrix multiplications and additions are complete. The master processing core 31 then reads the results and performs all remaining operations.

[0082] Figure 13 shows a possible arrangement of a five-core cluster to form a large-format image. In this case, the clusters at the edges of the array have a different arrangement of processing cores 31 due to geometric constraints. Two cores are unavailable and are represented by black squares. Note that the current configuration can be extended so that the master processing core 31 also performs the calculation of timestamps from the corresponding slave processing cores 31, creating an essentially uniform LSTM imager.

[0083] Processing cores 31 may also interoperate in a master-master configuration. For example, if an image is compressed by applying a function that involves using the current pixel value or TDC timestamp to look up a value in a lookup table. Each processing core 31 would need a copy of this table, but in the majority of cases the table would be too large to fit on each processing core 31; instead, the table could be split into portions and distributed to all processing cores 31. In this way, each processing core 31 can process its own input, but if the input is outside its range, it will send it to a neighboring processing core 31 that stores its respective portion of the table.

[0084] Any processing core can only send data to its neighboring processing core, but data can be further forwarded from neighboring processing cores to their neighbors. In this way, information can be shared between any two cores, but indirectly and slowly, since there is no direct physical connection between the two cores.

Claims

1. 1. An image sensor device in a stacked arrangement, comprising: a pixel array layer including a plurality of pixel segments each having a plurality of pixels for photon detection and each providing a digital pixel output; a processing layer including a number of processing cores each associated with one of a plurality of pixel segments and receiving pixel outputs of the pixels of the respective pixel segment, each processing core in bidirectional communication with one or more adjacent processing cores; 1. An image sensor device, wherein each processing core is configured to receive pixel outputs for pixels of an associated pixel segment, and to distribute processing of the pixel outputs between the processing core and at least one adjacent processing core.

2. 2. The image sensor device of claim 1, wherein the pixels of the pixel array layer include a detector diode, in particular a SPAD (Single-Photon Avalanche Diode), and a pre-processing circuit coupled to the detector diode and providing a respective pixel output as a signal, in particular the signal being provided to the processing layer via a through via in the pixel array layer.

3. 3. The image sensor device of claim 1, wherein each processing core comprises a front-end block comprising combinatorial logic and / or at least one look-up table that is freely configurable to provide masking and / or logical operations on the pixel outputs to pre-process the pixel outputs.

4. 4. The image sensor unit of claim 3, wherein a timing block is configured to receive the pre-processed pixel outputs and perform various timing functions such as pulse width and / or phase shift measurements.

5. 5. An image sensor arrangement according to claim 1, wherein a processing block is provided which includes a set of general purpose registers, an arithmetic and logic unit (ALU) and a RAM, and a control block is provided for controlling the operation of the processing block.

6. 6. An image sensor device as described in any one of claims 5, wherein a control block of at least one processing core associated with each pixel segment is configured to divide one or more processing tasks to be performed on pixels of that one pixel segment into processing portions, at least two processing portions being processed simultaneously in parallel, and the at least two processing portions being executed at least partially in parallel in the processing core and at least one adjacent processing core by directly controlling the processing of each processing portion within the processing block of the processing core and instructing at least one adjacent processing core to execute the other of the processing portions.

7. The processing tasks include matrix operations and / or addition operations performed on the image detected by the image sensor device, and in particular include LSTM operations including matrix operations, addition operations, sigmoid operations, and tangent hyperbolic operations, and the control blocks of each of the processing cores respectively: - dividing the execution of the matrix and addition operations to be performed on the pixels of said associated pixel segment into a plurality of processing parts, at least two of said processing parts being capable of operating simultaneously in parallel; - executing at least one of the processing parts in the control block; 7. An image sensor device as described in any one of claims 6, configured to be able to communicate at least one of the plurality of processing portions to at least one adjacent processing core adjacent to each processing core and instruct at least one of the adjacent processing cores to execute at least one of the plurality of processing portions.

8. 8. An image sensor device according to claim 1, wherein the processing cores associated with pixel segments at the edges of the pixel array are in bidirectional communication with one or more registers, respectively.

9. 9. The image sensor device of claim 1, wherein the processing cores communicate bidirectionally via a bidirectional data bus and a bidirectional timing signal line, respectively.

10. 10. An image sensor device according to any preceding claim, wherein the processing layer and the pixel array layer are formed on separate substrates that are stacked to form the image sensor device.

11. 11. The image sensor device of claim 1, wherein the processing tasks are separated into processing portions and at least one of the processing cores is configured to distribute the processing portions between at least one of the processing cores and at least one of the adjacent processing cores adjacent to the at least one of the processing cores.

Images