A single-photon tof image sensor for lidar

Abstract

The application discloses a kind of single-photon TOF image sensors for laser radar, including reconfigurable pixel array, dynamic reconfigurable circuit, real-time feedback control system;Reconfigurable pixel array is constructed based on field programmable array FPGA technology, each pixel unit is integrated with configurable single-photon avalanche diode SPAD, amplifier, integrator and control logic circuit, by programming to the internal logic of FPGA, change pixel operating mode and parameter, and pixel unit can be configured as unit with specific function, pixel for edge detection and motion detection, in the application, flexible reconfigurable pixel configuration: with field programmable array FPGA technology, the photosensitive area of pixel unit, integration time and gain and other key parameters can be flexibly adjusted according to actual demand, pixel unit can also be configured as unit with specific function such as edge detection, motion detection.

Description

Technical Field

[0001] This invention relates to the field of image sensor technology, and more particularly to a single-photon TOF image sensor for lidar. Background Technology

[0002] Time-of-flight (TOF) image sensors are a depth-sensing technology based on the principle of light pulse time-of-flight, primarily used to measure the distance between an object and the sensor. Its core principle involves emitting light pulses and calculating the time difference between their emission, reflection, and return to the sensor. This time difference, combined with the speed of light, is used to calculate the distance. Key technologies include high-precision time measurement, efficient optical signal processing, and low-noise photoelectric conversion. In recent years, with advancements in semiconductor technology and signal processing algorithms, the resolution, accuracy, and response speed of TOF sensors have significantly improved, while power consumption and cost have gradually decreased. Furthermore, TOF sensors are less dependent on lighting conditions and can operate stably in complex environments.

[0003] From the perspective of existing technologies, the pixel configuration of TOF image sensors lacks flexibility: most existing sensors have fixed pixel unit parameters, making it impossible to dynamically adjust the photosensitive area, integration time, and gain according to actual application scenarios. For example, in low-light environments, the inability to increase the photosensitive area and extend the integration time results in insufficient capture capability for weak light and poor image quality. Moreover, they typically lack the ability to configure pixel units as specific functional units, making it difficult to meet diverse task requirements and limiting application scenarios. They are only suitable for simple scenarios with simple pixel characteristic requirements and minimal environmental changes. Furthermore, many existing sensors can only collect single types of data, such as optical imaging, lacking multimodal fusion interfaces. Even if some sensors have multimodal acquisition capabilities, the lack of effective fusion algorithms prevents them from fully exploring the correlation between different modal data and deeply analyzing the comprehensive characteristics of the target object. This results in one-sided environmental information acquired in complex scenarios, leading to low accuracy and reliability in detection and recognition. Therefore, a single-photon TOF image sensor for LiDAR is proposed. Summary of the Invention

[0004] To overcome the aforementioned shortcomings of the prior art, embodiments of the present invention provide a single-photon TOF image sensor for lidar, which lacks flexibility in pixel configuration and does not have the ability to configure pixel units as specific functional units, making it difficult to meet diverse task requirements and limiting application scenarios.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A single-photon TOF image sensor for lidar includes a reconfigurable pixel array, a dynamically reconfigurable circuit, and a real-time feedback control system.

[0007] The reconfigurable pixel array is built on field-programmable array (FPGA) technology. Each pixel unit integrates a configurable single-photon avalanche diode (SPAD), an amplifier, an integrator, and control logic circuitry. By programming the internal logic of the FPGA, the pixel's operating mode and parameters can be changed, and the pixel unit can be configured as a unit with specific functions for edge detection and motion detection.

[0008] The dynamically reconfigurable circuit includes a time-to-digital converter (TDC) and a data processing circuit. The time resolution and measurement range of the TDC are dynamically adjusted according to the actual application scenario. The data processing circuit adopts a reconfigurable architecture, and its internal computing and storage units can be flexibly configured according to different application scenarios and algorithm requirements. Furthermore, the TDC and the data processing circuit work closely together to adjust the working parameters and collaboration methods in real time according to the characteristics of the data collected by the sensor and the application requirements.

[0009] The real-time feedback control system is used to analyze the data collected by the sensors in real time. Based on the analysis results, it automatically generates control signals and sends them to the FPGA to realize the dynamic reconstruction of the sensor hardware. The system forms a closed-loop optimization mechanism, continuously monitors the sensor performance and data quality, and further adjusts the configuration parameters according to the evaluation results until the sensor performance meets the application requirements.

[0010] Preferably, in the reconfigurable pixel array, the single-photon avalanche diode (SPAD) employs quantum dot enhancement technology, achieving a quantum efficiency ≥85%@550nm and a dark count rate ≤100cps / mm. 2 The pixel unit supports dynamic adjustment of the photosensitive area by 0.5μm. 2 -100μm 2 Integration time adjustment (1μs-1s) and gain control (10dB-60dB)

[0011] Edge detection pixels integrate a dedicated circuit for gradient operators, employing the Sobel operator to calculate the gradient of the image in the x and y directions, with the gradient G in the x direction being... x (i,j), the calculation formula is:

[0012]

[0013] in Let I be the Sobel operator in the x-direction, (i,j) be the image matrix, (i,j) be the coordinates of a pixel in the image, and m and n be the offsets of the Sobel operator matrix elements; and G be the gradient in the y-direction. y (i,j), the calculation formula is:

[0014]

[0015] in The Sobel operator in the y-direction has an edge strength of Edge direction And the response time is <50ns;

[0016] Motion detection pixels, based on the temporal difference algorithm, let the current frame image be I. t (x,y), I t-Δt (x,y), then the time is

[0017] ΔI(x,y)=I t (x,y)-I t-Δt (x,y),

[0018] By setting a threshold T, when ΔI(x,y)>T, it is considered that the pixel has motion, supporting 100fps dynamic tracking.

[0019] Preferably, in the dynamically reconfigurable circuit, the TDC uses interpolation time measurement technology to achieve dynamically adjustable time resolution of 0.1ps-1ns and adaptive expansion of measurement range of 0mm-10km, supporting 128 parallel measurements; the data processing circuit adopts a heterogeneous computing architecture, including a general-purpose processing kernel, a convolutional neural network accelerator and a three-level storage architecture, supporting dynamic voltage and frequency adjustment of 0.6V-1.2V, 100MHz-2GHz;

[0020] The TDC interpolation algorithm: Let the coarse counting time be t. coarse The fine voltage measurement value is V fine The reference voltage is V. ref The clock period is T clk The measurement time is

[0021]

[0022] Data processing circuit deep learning inference algorithm: Taking a fully connected layer as an example, let the input vector be x, the weight vector be W, the bias vector be b, and the activation function be σ, then the output vector...

[0023] y = σ(Wx + b);

[0024] For a convolutional layer, let the input feature map be X, the convolution kernel be K, and the bias be b, then the convolution output...

[0025] Y = Conv(X,K) + b.

[0026] Preferably, the real-time feedback control system integrates a deep learning-based performance evaluation model and an adaptive control algorithm, achieves closed-loop optimization through a hardware configuration parameter database, and supports multimodal data fusion, including: a radio frequency signal processing interface, a multispectral detector interface, and an inertial measurement unit interface;

[0027] In an adaptive control algorithm, the error signal is given by e(t) = r(t) - y(t), where r(t) is the setpoint and y(t) is the actual output value. The control output... Where K p It is the proportionality coefficient, K i It is the proportionality coefficient, K d These are differential coefficients;

[0028] In a deep learning algorithm, given state s, action selection a, reward r, learning rate α, and discount factor γ, the value of Q is...

[0029] Q(s t ,a t )=Q(s t ,a t )+α[r t +γmax a Q(s t+1 ,a)-Q(s t ,a t )];

[0030] Multimodal data fusion algorithms, based on Bayes' theorem, consider events A and B. The probability of A occurring given that B has occurred is... In multimodal data fusion, A represents a certain attribute of the target, and B represents data from different modal sensors.

[0031] Preferably, the pixel array adopts a three-dimensional stacked architecture to achieve vertical integration of the pixel layer (65nm), circuit layer (28nm), and storage layer (16nm), reducing the chip area by 40%.

[0032] Preferably, the TDC employs quantum dot effect ultra-fine measurement technology for close-range measurements to achieve picosecond-level resolution; for long-range measurements, it uses signal enhancement algorithms combined with optimized amplification and transmission circuits to achieve a measurement accuracy of ±0.1%.

[0033] Preferably, the data processing circuit includes:

[0034] Simple task mode: Configured to low power consumption of 10mW-50mW, with a processing speed of up to 100fps;

[0035] Complex task mode: Activate the dedicated accelerator, with computing power reaching 10 TOPS, supporting real-time deep learning inference.

[0036] Preferably, the real-time feedback control system includes:

[0037] Light intensity change response: Dynamic adjustment from 0.1 lux to 100,000 lux within 50 ms;

[0038] Moving target tracking: Based on the Kalman filter algorithm, trajectory prediction error <5cm / s;

[0039] Multimodal fusion: Optical, radio frequency, and inertial data are fused through Bayesian networks, achieving a target material recognition accuracy of ≥98%.

[0040] Preferably, the sensor includes:

[0041] Distance measurement mode: Automatic switching within the range of 0.1mm-10km, with sub-millimeter accuracy <10m;

[0042] Operating temperature range: -40℃ to 125℃, using package-level temperature compensation technology.

[0043] Preferably, the sensor integration includes:

[0044] On-chip calibration module: supports automatic dark current compensation and photon response non-uniformity correction;

[0045] Safety protection mechanisms: overvoltage protection ±20V, ESD protection HBM4kV.

[0046] The technical effects and advantages of the single-photon TOF image sensor for lidar in this invention are as follows:

[0047] 1. This invention features flexible and reconfigurable pixel configuration: With the help of field-programmable array (FPGA) technology, key parameters such as the photosensitive area, integration time, and gain of the pixel unit can be flexibly adjusted according to actual needs. The pixel unit can also be configured to have specific functions such as edge detection and motion detection. This highly flexible configuration method enables the sensor to quickly adapt to different application scenarios and task requirements, greatly improving its wide range of applications and adaptability.

[0048] 2. This invention possesses powerful multimodal data fusion capabilities, integrating a multimodal fusion interface to support access to radio frequency signal processing modules, multispectral detectors, and inertial measurement units. Through algorithms such as Bayesian networks, it efficiently fuses sensor data from different modalities, deeply analyzing the material, shape, position, and motion state of target objects. This advantage enables sensors to acquire more comprehensive and accurate environmental information, providing stronger data support for applications in complex scenarios and significantly improving the accuracy and reliability of detection and identification.

[0049] 3. This invention features efficient real-time feedback control: It incorporates a deep learning-based performance evaluation model and adaptive control algorithm, and achieves closed-loop optimization through a hardware configuration parameter database. The real-time feedback control system can analyze the data collected by the sensor in real time, automatically generate control signals based on the analysis results and send them to the FPGA, quickly realize the dynamic reconstruction of the sensor hardware, ensure that the sensor is always in the best working state, and provide stable and high-quality data output for various applications.

[0050] 4. This invention employs a Time-to-Digital Converter (TDC) using interpolation time measurement technology, achieving dynamic adjustment of time resolution and adaptive expansion of the measurement range, thus meeting measurement tasks with varying distances and accuracy requirements. The data processing circuit adopts a heterogeneous computing architecture, including a general-purpose processing core, dedicated acceleration modules, and a three-level storage architecture, supporting dynamic voltage and frequency adjustment. This high-performance hardware architecture not only ensures efficient and accurate data processing but also allows for flexible adjustment of computing resources and power consumption based on task complexity and real-time requirements, achieving an optimal balance between performance and energy consumption. Attached Figure Description

[0051] Figure 1 This is a system module block diagram of a single-photon TOF image sensor for lidar proposed in this invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0053] Example 1

[0054] refer to Figure 1This embodiment provides an implementation of a reconfigurable pixel array in a single-photon Time-of-Flight (TOF) image sensor for LiDAR, and the specific implementation steps include:

[0055] 1. Hardware Design

[0056] (1) FPGA selection: FPGA has abundant logic resources, high-speed data processing capabilities and powerful parallel processing characteristics, which can meet the needs of complex logic control and high-speed data processing of pixel units, and easily realize the configuration and management of a large number of pixel units.

[0057] (2) SPAD design and integration: Using advanced semiconductor technology, quantum dot-enhanced SPADs are manufactured in a 65nm process. Quantum dot materials are precisely integrated into the photosensitive area of ​​the SPAD through photolithography and deposition processes to improve its quantum efficiency and reduce dark count rate. The SPAD in each pixel unit is connected to the amplifier, integrator and control logic circuit through metal wiring to ensure fast signal transmission and processing.

[0058] (3) Pixel unit circuit layout: In the chip layout design, the amplifier, integrator and control logic circuit are compactly arranged around the SPAD to reduce signal transmission delay. Multi-layer metal wiring technology is used to optimize circuit connection and improve chip integration and reliability. For example, high-speed data communication between pixel units is realized through the top metal layer, and the bottom metal layer is responsible for power supply and grounding.

[0059] 2. Software programming

[0060] (1) Pixel mode configuration algorithm: In the programmable logic of the FPGA, a pixel mode configuration algorithm is written. By parsing the configuration instructions input by the user, the corresponding control signals are generated to realize the dynamic adjustment of the photosensitive area, integration time and gain of the pixel unit. For example, when the user needs to increase the pixel sensitivity, the algorithm will increase the area of ​​the photosensitive area and extend the integration time, while adjusting the gain setting of the amplifier.

[0061] (2) Edge detection pixel implementation: The gradient algorithm of edge detection pixels is implemented using the hardware description language of FPGA. The calculation logic of Sobel operator is written as a hardware module. The calculation speed is improved by pipeline design. Each edge detection pixel unit runs this module to process the input image signal in real time and quickly output edge intensity and direction information.

[0062] (3) Implementation of motion detection pixels: The time difference algorithm for motion detection pixels is also implemented using hardware description language; a frame buffer module is built inside the FPGA to store the current frame and the previous frame image data. The two frames of image data are subtracted pixel by pixel through hardware logic and compared with the preset threshold to determine whether the pixel has moved; for the motion detection pixel array, a parallel processing architecture is adopted to realize fast motion detection of the entire image area.

[0063] Example 2

[0064] refer to Figure 1 This embodiment provides an implementation of a dynamically reconfigurable circuit in a single-photon Time-of-Flight (TOF) image sensor for LiDAR, and the specific implementation steps include:

[0065] 1. TDC Hardware

[0066] (1) Interpolation TDC circuit design: The interpolation TDC circuit structure based on delay line is adopted. High-precision delay units are used to form delay lines. Through precise control and measurement of delay time, dynamic adjustment of time resolution is achieved. For example, aluminum nitride-based surface acoustic wave delay lines are used, which have extremely low temperature coefficient and high stability, and can provide accurate delay in the time resolution range of 0.1ps-1ns.

[0067] (2) TDC measurement range extension circuit: In order to achieve adaptive extension of the measurement range from 0mm to 10km, a signal amplification and attenuation circuit is designed. When measuring at close range, the signal strength is reduced by the attenuation circuit to avoid TDC saturation. When measuring at long distance, a multi-stage amplifier is used to amplify the signal to improve the detectability of the signal. At the same time, in conjunction with the automatic gain control (AGC) circuit, the amplification factor is automatically adjusted according to the signal strength to ensure that the TDC can work accurately in different measurement ranges.

[0068] (3) Multi-channel TDC integration: Design TDC channels, each working independently to achieve simultaneous measurement of multiple targets or different positions of the same target; adopt time division multiplexing technology, and transmit the signals of different channels to the subsequent data processing circuit in sequence through high-speed switching circuit to improve the measurement efficiency and data processing capability of the system.

[0069] 2. Data processing power hardware

[0070] (1) Implementation of heterogeneous computing architecture: Select a general-purpose processing kernel to be responsible for the overall control of the system, task scheduling and simple data processing tasks; at the same time, integrate a specially designed convolutional neural network accelerator as a dedicated acceleration module for fast computation of deep learning algorithms; for example, adopt a convolutional neural network accelerator based on field-programmable array (FPGA) to utilize the parallel computing capabilities of FPGA to perform hardware acceleration of operations such as convolution and pooling, which can achieve several times the performance improvement compared to general-purpose processors;

[0071] (2) Three-level storage architecture construction: Construct a three-level storage architecture consisting of register file, cache and DRAM; register file is located inside the processor core and is used to store the critical data being processed, providing the fastest data access speed; cache uses static random access memory (SRAM) to cache frequently used data and instructions, reduce the number of times DRAM is accessed, and improve data processing efficiency; DRAM, as a large-capacity main memory, stores a large amount of image data and algorithm parameters; through reasonable cache management algorithms, efficient data transfer and storage between the three levels of storage are achieved;

[0072] (3) Dynamic Voltage and Frequency Adjustment Circuit: A Dynamic Voltage and Frequency Adjustment (DVFS) circuit is designed to automatically adjust the operating voltage and frequency of the processor core and dedicated acceleration modules by monitoring the load of data processing tasks. For example, when performing simple image filtering tasks, the operating voltage and frequency are reduced to reduce power consumption; when performing complex deep learning inference, the operating voltage and frequency are increased to improve the computing speed. The DVFS circuit works in conjunction with the power management chip to achieve precise control of system power consumption.

[0073] 3. Data processing circuit software

[0074] (1) Deployment of deep learning inference algorithm: The deep learning inference algorithm is converted into a hardware executable instruction set by the compiler and deployed to the convolutional neural network accelerator. Under the control of the kernel, the input image data is preprocessed, and then the processed data is transmitted to the accelerator for deep learning inference. The inference results are then post-processed and analyzed by the ARM kernel to realize the recognition and classification of target objects.

[0075] (2) Switching between simple task mode and complex task mode: Write a task scheduling algorithm to automatically switch the working mode of the data processing circuit according to the type and complexity of the data processing task; when a simple image filtering or edge detection task is received, the algorithm configures the data processing circuit to simple task mode, shuts down the dedicated accelerator, and uses only the ARM core for processing to reduce power consumption; when a complex deep learning task is encountered, the dedicated accelerator is started, the access strategy of the storage architecture is adjusted, the data transmission path is optimized, and the computing speed and processing efficiency are improved.

[0076] Example 3

[0077] refer to Figure 1 This embodiment provides an implementation of a real-time feedback control system in a single-photon Time-of-Flight (TOF) image sensor for lidar, and the specific implementation steps include:

[0078] 1. Hardware Design

[0079] (1) Hardware acceleration module for performance evaluation model: In order to improve the computing speed of deep learning-based performance evaluation model, a dedicated hardware acceleration module is designed. The key computing units in the model, such as convolutional layers and fully connected layers, are implemented using field-programmable arrays (FPGAs) or application-specific integrated circuits (ASICs). Through hardware acceleration, the data collected by the sensors can be analyzed and evaluated in real time, providing a basis for subsequent control decisions.

[0080] (2) Hardware implementation of adaptive control algorithm: The key computational parts of the PID control algorithm and reinforcement learning algorithm are programmed into hardware modules using a hardware description language and integrated into the real-time feedback control system. The parallel computing capabilities of the hardware are utilized to improve the execution speed of the control algorithm and enable rapid adjustment of sensor hardware parameters;

[0081] (3) Hardware configuration parameter database storage: A high-speed flash memory chip is used to store the hardware configuration parameter database, which contains configuration templates; it is connected to the FPGA through a high-speed interface circuit to ensure that configuration parameters can be quickly read and written during system operation, thereby realizing dynamic reconfiguration of sensor hardware.

[0082] 2. Software programming implementation

[0083] (1) Training and application of deep learning-based performance evaluation model: A large amount of sensor data is used to train the deep learning-based performance evaluation model. The training data includes image data, sensor measurement data and corresponding performance indicators in different scenarios. After training, the model is deployed to the real-time feedback control system. During system operation, the model evaluates the performance of the sensors in real time based on the data collected by the sensors, such as measurement accuracy, image quality, power consumption, etc., and feeds the evaluation results back to the adaptive control algorithm.

[0084] (2) Integration of adaptive control algorithm and reinforcement learning algorithm: Write software program to integrate adaptive control algorithm and reinforcement learning algorithm; In the initial stage of the system, the adaptive control algorithm is mainly used to adjust the sensor hardware parameters so that the system can quickly reach a stable state; As the system runs, the reinforcement learning algorithm continuously optimizes the control strategy according to the sensor performance feedback and environmental changes, and gradually dominates the control process. Through the integration of the two, the adaptive adjustment of sensor hardware parameters is realized, and the performance of the sensor in different scenarios is improved.

[0085] (3) Implementation of multimodal data fusion algorithm: The Bayesian network multimodal data fusion algorithm is implemented by software programming; during system operation, data from the radio frequency signal processing module, multispectral detector and inertial measurement unit are collected in real time. These data are used as inputs to the Bayesian network. Through the calculation of Bayes' theorem, data from different modes are fused to obtain more accurate information about the target object, such as the material, position and motion state of the target object. The fusion results are used for sensor performance evaluation and control decision.

[0086] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0087] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0088] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that cannot be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

[0089] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A single-photon TOF image sensor for a lidar, characterized by The reconfigurable pixel array, the dynamic reconfigurable circuit, the real-time feedback control system; The reconfigurable pixel array is constructed based on field programmable array (FPGA) technology, each pixel unit is integrated with a configurable single photon avalanche diode (SPAD), an amplifier, an integrator and a control logic circuit, the working mode and parameters of the pixel unit can be changed by programming the internal logic of the FPGA, and the pixel unit can be configured as a unit with a specific function for edge detection and motion detection; The dynamic reconfigurable circuit includes a time-to-digital converter (TDC) and a data processing circuit, the time resolution and measurement range of the TDC are dynamically adjusted according to the actual application scenario, the data processing circuit adopts a reconfigurable architecture, the internal operation unit and storage unit of which can be flexibly configured according to different application scenarios and algorithm requirements, and the TDC and the data processing circuit closely cooperate, and the working parameters and cooperation mode are adjusted in real time according to the data characteristics collected by the sensor and the application requirements; The real-time feedback control system is used for real-time analysis of the data collected by the sensor, generates a control signal based on the analysis result and sends the control signal to the FPGA to realize dynamic reconfiguration of the sensor hardware, the system forms a closed-loop optimization mechanism, continuously monitors the performance of the sensor and the quality of the data, and further adjusts the configuration parameters according to the evaluation result until the performance of the sensor meets the application requirements.

2. A single-photon TOF image sensor for a lidar as claimed in claim 1, characterized in that, In the reconfigurable pixel array, a single photon avalanche diode (SPAD) adopts quantum dot enhancement technology, quantum efficiency ≥ 85% at 550nm, dark count rate ≤ 100cps / mm 2 ; the pixel unit supports dynamic adjustment of the photosensitive area 0.5μm 2 -100μm 2 , integration time adjustment 1μs-1s and gain control 10dB-60dB, Edge detection pixels, integrated gradient operator dedicated circuit, using Sobel operator to calculate the gradient of image in x and y direction, x direction gradient G x (i,j), the formula is wherein is the Sobel operator in the x direction, I is the image matrix, (i,j) is the coordinate of the pixel in the image, and m and n are the offsets of the Sobel operator matrix elements; the y direction gradient G y (i,j), and the calculation formula is wherein Sobel operator in the y direction, the edge intensity is edge direction and the response time is < 50 ns; Motion detection pixels, based on a temporal difference algorithm, set the current frame image as I t (x,y), I t-Δt (x,y), the time is ΔI(x,y) = I t (x,y) - I t-Δt (x,y), By setting a threshold T, when ΔI(x, y) > T, it is considered that motion occurs at the pixel point, and 100fps dynamic tracking is supported.

3. The single-photon TOF image sensor for a lidar of claim 1, wherein, In the dynamic reconfigurable circuit, the TDC adopts interpolation time measurement technology, realizes dynamic adjustment of time resolution of 0.1ps-1ns and adaptive expansion of measurement range of 0mm-10km, and supports 128 parallel measurements; the data processing circuit adopts a heterogeneous computing architecture, includes a general processing core, a convolutional neural network accelerator and a three-level storage architecture, supports dynamic voltage frequency adjustment of 0.6V-1.2V and 100MHz-2GHz; Wherein the TDC interpolation algorithm: let the coarse count time be t coarse The fine voltage measurement value is V fine , the reference voltage is V ref , the clock period is T clk , and the measurement time is The deep learning inference algorithm of the data processing circuit: taking a fully connected layer as an example, assuming that the input vector is x, the weight vector is W, the bias vector is b, and the activation function is σ, the output vector is y = σ(Wx + b); For a convolutional layer, assuming that the input feature map is X, the convolution kernel is K, and the bias is b, the convolution output is Y = Conv(X, K) + b.

4. The single-photon TOF image sensor for a lidar of claim 1, wherein, The real-time feedback control system integrates a performance evaluation model based on deep learning and an adaptive control algorithm, realizes closed-loop optimization through a hardware configuration parameter database, and supports multi-modal data fusion, including a radio frequency signal processing interface, a multi-spectral detector interface and an inertial measurement unit interface; An adaptive control algorithm, with error signal e(t) = r(t) - y(t), where r(t) is the setpoint and y(t) is the actual output value, the control output where K p is a proportional coefficient, K i is a proportional coefficient, K d is a derivative coefficient; The deep learning algorithm assumes that the state is s, the action selection is a, the reward is r, the learning rate is α, and the discount factor is γ, then the value of Q is Q(s t ,a t ) = Q(s t ,a t ) + a[r t + y max a Q(s t+1 ,a) - Q(s t ,a t )]; A multi-modal data fusion algorithm, according to Bayes' theorem, let event A and B, then the probability of A occurring under the condition that B has occurred In multi-modal data fusion, A represents some attribute of the target, and B represents the data of different modal sensors.

5. The single-photon TOF image sensor for a lidar of claim 1, wherein, The pixel array adopts a three-dimensional stacked architecture, realizes vertical integration of a pixel layer of 65nm, a circuit layer of 28nm and a storage layer of 16nm, and reduces the chip area by 40%.

6. The single-photon TOF image sensor for a lidar of claim 1, wherein, The TDC adopts quantum dot effect super-fine measurement technology for near distance measurement, achieving picosecond level resolution; and through signal enhancement algorithm, combined with optimized amplification and transmission circuit, the measurement accuracy reaches ±0.1% for long distance measurement.

7. The single-photon TOF image sensor for a lidar of claim 1, wherein, The data processing circuit comprises: Simple task mode: configured as a low-power state of 10-50 mW, and the processing speed reaches 100 fps; Complex task mode: a special accelerator is activated, the computing power reaches 10 TOPS, and real-time deep learning inference is supported.

8. The single-photon TOF image sensor for a lidar of claim 1, wherein, The real-time feedback control system comprises: Light mutation response: 0.1 lux-100,000 lux dynamic adjustment is completed within 50 ms; Motion target tracking: based on Kalman filtering algorithm, the trajectory prediction error is less than 5 cm / s; Multi-modal fusion: through a Bayesian network, optical, radio frequency and inertial data fusion is realized, and the target material identification accuracy is greater than or equal to 98%.

9. The single-photon TOF image sensor for a lidar of claim 1, wherein, The sensor comprises: Distance measurement mode: automatic switching in the range of 0.1 mm-10 km, sub-millimeter level precision less than 10 m; Working temperature range: -40-125 DEG C, and a packaging level temperature compensation technology is adopted.

10. The single-photon TOF image sensor for a lidar of claim 1, wherein, The sensor integration comprises: On-chip calibration module: supporting automatic dark current compensation and photon response non-uniformity correction; Safety protection mechanism: overvoltage protection ±20 V, ESD protection HBM 4 kV.

Images