Visual imaging and recognition device

Abstract

The present disclosure provides a visual imaging and recognition device, which can be applied to the field of image processing technology. The device comprises: an imaging sensor for acquiring two-dimensional image information and three-dimensional depth information of a target object; a signal shaping and coding module for signal shaping and coding the two-dimensional image information and the three-dimensional depth information to obtain target two-dimensional image information and target three-dimensional depth information; a processing module for processing and recognizing the target two-dimensional image information and the target three-dimensional depth information to generate a recognition result; an output module for signal synchronous buffering and outputting the recognition result; and a programmable microcontroller for providing synchronous control signals to the imaging sensor, the signal shaping and coding module, the processing module and the output module to control the synchronous and coordinated operation of the device. The device can naturally imitate the working mechanism of the human brain visual system and has the advantages of high performance and low power consumption.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a visual imaging and recognition device. Background Technology

[0002] With the continuous development of information technology integration and application, edge-side intelligent devices have been widely used in fields such as intelligent security, robot vision, intelligent assisted driving, and consumer electronics, becoming one of the important directions for the future development of electronic information technology. Edge-side intelligent devices refer to devices located at the edge of the data network. Unlike data network-centric devices, edge-side intelligent devices have higher requirements for power consumption, timeliness, security, and integration. For example, the power consumption performance of a device directly determines its usage time, and the data transmission and computing latency performance directly determines its processing timeliness. Therefore, low power consumption and low latency design have always been important design goals for edge devices. Visual information acquisition and processing systems are a typical example of edge-side intelligent devices, capable of independently acquiring and intelligently processing image information. Traditional visual information acquisition and processing systems employ a separate structure for sensing, storage, and processing, resulting in limited bandwidth for large amounts of sensor data transmission between the sensor and the processor. The data transmission speed bottleneck between the memory and the processor also limits system performance. This separate structure fundamentally fails to solve the system latency and energy consumption problems caused by the handling of large amounts of sensor data. The integrated visual information acquisition and processing system combines image sensors, intelligent processors, and memory into one integrated system, which can efficiently realize the functions of image signal acquisition, storage, and intelligent processing, and has become one of the important research topics in the field of semiconductor information today. Summary of the Invention

[0003] (a) Technical problems to be solved

[0004] To address at least one of the aforementioned technical problems in existing visual imaging and recognition devices, embodiments of this disclosure provide a visual imaging and recognition device that can be used for imaging in low-light environments. It features the ability to fuse two-dimensional image information and three-dimensional depth information, enabling rapid target recognition and depth information reconstruction of target objects with extremely low power consumption. This device can be used to determine whether the object under test is a three-dimensional object or a two-dimensional photograph, and can be applied to scenarios such as intelligent security, facial recognition, robot vision, and intelligent assisted driving.

[0005] (II) Technical Solution

[0006] To address the aforementioned technical problems, embodiments of this disclosure propose a visual imaging and recognition device.

[0007] According to a first aspect of this disclosure, a visual imaging and recognition device is provided, comprising: an imaging sensor for acquiring two-dimensional image information and three-dimensional depth information of a target object; a signal shaping and encoding module for performing signal shaping and encoding on the two-dimensional image information and the three-dimensional depth information to obtain target two-dimensional image information and target three-dimensional depth information; a processing module for processing and recognizing the target two-dimensional image information and target three-dimensional depth information to generate a recognition result; an output module for synchronously buffering and outputting the recognition result; and a programmable microcontroller for providing synchronous control signals to the imaging sensor, the signal shaping and encoding module, the processing module, and the output module to control the synchronous and coordinated operation of the device.

[0008] In some exemplary embodiments, the imaging sensor includes a single-photon array imaging sensor.

[0009] In some exemplary embodiments, the two-dimensional image information includes pulsed two-dimensional image information, wherein the pulse frequency of the pulsed two-dimensional image information is positively correlated with the intensity of the light incident signal of the target object; and the method for measuring the three-dimensional depth information includes the time-of-flight method.

[0010] In some exemplary embodiments, the processing module includes: a non-volatile in-memory computing submodule, used to perform pulse convolutional neural network deep learning and processing on the target two-dimensional image information to achieve intelligent processing of the target two-dimensional image information and obtain the two-dimensional image recognition processing result in the analog domain; and a volatile in-memory computing submodule, used to perform programmable digital signal processing on the target three-dimensional depth information to achieve reconstruction and depth discrimination of the target three-dimensional depth information and obtain the three-dimensional image reconstruction and depth discrimination result in the digital domain, wherein the non-volatile in-memory computing submodule and the volatile in-memory computing submodule are heterogeneous and parallel, and are programmed and controlled by a programmable microcontroller.

[0011] In some exemplary embodiments, the processing module further includes a storage system, wherein the non-volatile in-memory computing submodule and the volatile in-memory computing submodule share the storage system.

[0012] In some exemplary embodiments, the non-volatile in-memory computing submodule includes a resistive variable memory cell array, wherein the resistive variable memory cells have non-volatile characteristics and in-memory computing characteristics; and / or the resistive variable memory cell array includes transistors and resistive variable resistors; and / or the input signals of the resistive variable memory cells include digital signals, and the output signals include current-mode analog signals.

[0013] In some exemplary embodiments, the volatile in-memory computing submodule includes a static random access memory (SRAM) cell array, wherein the SRAM cells are capable of performing logical operations within the SRAM cells; and / or the SRAM cells include transistors; and / or the input signals of the SRAM cells include digital signals, and the output signals include digital signals.

[0014] In some exemplary embodiments, the recognition results include two-dimensional image recognition processing results in the analog domain and three-dimensional image reconstruction and depth discrimination results in the digital domain; the output module is used to convert the two-dimensional image recognition processing results in the analog domain into two-dimensional image recognition processing results in the digital domain; and the output module is used to integrate the two-dimensional image recognition processing results in the digital domain with the three-dimensional image reconstruction and depth discrimination results in the digital domain and output them synchronously.

[0015] In some exemplary embodiments, the programmable microcontroller receives an input synchronization clock signal and provides synchronization control signals to the imaging sensor, signal shaping and encoding module, processing module and output module through the synchronization clock signal; and / or the programmable microcontroller supports the RISC-V architecture.

[0016] In some exemplary embodiments, the programmable microcontroller is bidirectionally connected to at least one of the following modules: the imaging sensor, the signal shaping and encoding module, the processing module, and the result output module, for transmitting control signals to the connection module and receiving feedback signals from the connection module.

[0017] (III) Beneficial Effects

[0018] As can be seen from the above technical solutions, the visual imaging and recognition device provided in this disclosure has at least the following beneficial effects:

[0019] (1) The characteristics of single-photon imaging and signal output are similar to the mechanism by which the human retina perceives images and outputs electrical pulse signals. The pulsed convolutional neural network integrates the characteristics of traditional convolutional neural networks and spiking neural networks, mimicking the way human visual information is transmitted and processed. Integrating the two functional modules into the same imaging and recognition device can naturally mimic the working mechanism of the human brain's visual system, and has the advantages of high performance and low power consumption.

[0020] (2) Resistive variable memory cells have non-volatile characteristics, which allows the processing module to retain neural network model parameters even after power failure, saving the delay time of input parameters after each power-on. In addition, the multiply-accumulate operation in the analog domain of the resistive variable memory cell has a higher energy efficiency ratio than the logical calculation in the digital domain.

[0021] (3) Both resistive random access memory (RRAM) cells and static random access memory (SRAM) cells have the characteristic of in-memory computation, which enables the processing module to perform multiplication and accumulation operations at extremely high speed and extremely low power consumption, eliminating the delay and energy consumption of data transmission between traditional discrete memory and processor.

[0022] (4) The programmable microcontroller supports the RISC-V architecture and can programmatically control the heterogeneous parallel computing of the non-volatile in-memory computing sub-module and the volatile in-memory computing sub-module in the in-memory computing heterogeneous parallel processing module. At the same time, the RISC-V architecture has the characteristics of low power consumption, low cost, flexible scalability and high security. It can be flexibly programmed and customized according to application requirements. It has a simplified instruction set and a compact kernel, and can achieve efficient instruction execution with extremely low power consumption.

[0023] (5) The single-photon array imaging sensor is used. The output pulse signal has the characteristics of sparsity, simultaneous output of two-dimensional image information and three-dimensional depth information, large imaging dynamic range, high sensitivity and integration with silicon-based chips.

[0024] (6) It can be used for imaging in low light environment, has the function of fusion processing of two-dimensional image information and three-dimensional depth information, can perform fast target recognition and depth information reconstruction of the object under test under extremely low power consumption, and can distinguish whether the object under test is a three-dimensional object or a two-dimensional photograph. It can be applied to intelligent security, face recognition, robot vision and intelligent assisted driving and other application scenarios. Attached Figure Description

[0025] Figure 1 A schematic diagram illustrating the structure of a visual imaging and recognition device according to an embodiment of the present disclosure is shown; and

[0026] Figure 2 A schematic diagram of the structure of a processing module according to an embodiment of the present disclosure is shown.

[0027] Explanation of reference numerals in the attached figures: 100 - Visual imaging and recognition device; 10 - Imaging sensor; 20 - Signal shaping and encoding module; 30 - Processing module; 31 - Non-volatile in-memory computing submodule; 311 - Resistive variable memory unit; 32 - Volatile in-memory computing submodule; 321 - Static random access memory unit; 33 - Storage system; 40 - Output module; 50 - Programmable microcontroller. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0029] Figure 1 The schematic diagram illustrates the structure of a visual imaging and recognition device 100 according to an embodiment of the present disclosure.

[0030] like Figure 1 As shown, a visual imaging and recognition device 100 according to an embodiment of this disclosure includes: an imaging sensor 10 for acquiring two-dimensional image information and three-dimensional depth information of a target object; a signal shaping and encoding module 20 for performing signal shaping and encoding on the two-dimensional image information and the three-dimensional depth information to obtain target two-dimensional image information and target three-dimensional depth information; a processing module 30 for processing and recognizing the target two-dimensional image information and target three-dimensional depth information to generate a recognition result; an output module 40 for synchronously buffering and outputting the recognition result; and a programmable microcontroller 50 for providing synchronous control signals to the imaging sensor 10, the signal shaping and encoding module 20, the processing module 30, and the output module 40 to control the device to operate synchronously and in coordination.

[0031] In some exemplary embodiments, the imaging sensor 10 includes a single-photon array imaging sensor. The input terminal of the single-photon array imaging sensor serves as the input terminal of the visual imaging and recognition device 100, and the input signal is the light incident signal of the object under test. Optionally, the single-photon array imaging sensor is composed of single-photon avalanche diodes arranged in an array. Utilizing the characteristic that single-photon avalanche diodes can detect light signals at the level of a single photon in Geiger mode and output photon detection pulse signals, it completes the acquisition of two-dimensional light intensity image information and three-dimensional depth image information, realizing image sensing function. The pulse frequency of the pulsed two-dimensional image information is proportional to the light incident signal intensity of the object under test, and the three-dimensional depth information is obtained by measuring the time-of-flight method. The time-of-flight method includes converting the acquired image into a series of pulse signals when it is detected. To obtain the three-dimensional depth information of the imaged target, the detection signal of the single-photon imaging needs to be statistically analyzed over time, and the time of flight from the emission of the light pulse from the light source to the reflection of the object under test and detected by the single-photon pixel is measured. The time-of-flight method is used to measure the depth information of the imaged object. The electrical signals caused by each light pulse are accumulated, and three-dimensional depth information is obtained through a suitable number of accumulations. The advantages of single-photon array imaging sensors include the sparsity of the output pulse signal, the ability to simultaneously output two-dimensional image information and three-dimensional depth information, a large imaging dynamic range, high sensitivity, and the ability to be integrated with silicon-based chips.

[0032] Based on the principle of single-photon imaging, a single-photon array imaging sensor, composed of a single-photon array, can generate a pulse frequency modulation signal related to the intensity of incident light on the object under test. The higher the light intensity of the image under test, the higher the frequency of the electrical pulses generated by the imaging sensor 10. Furthermore, it can simultaneously detect two-dimensional image information and three-dimensional depth information. These characteristics conform to the image information perception method of the human retina, generating pulse electrical signals related to the light intensity of the acquired image and obtaining an object image with depth information. Therefore, using a single-photon array imaging sensor as the imaging sensor 10 can reduce the amount of output data, reduce power consumption, and improve imaging sensitivity and dynamic range.

[0033] In some exemplary embodiments, the processing module 30 includes a non-volatile in-memory computing submodule 31 and a volatile in-memory computing submodule 32. The non-volatile in-memory computing submodule 31 performs pulsed convolutional neural network deep learning and processing on the target two-dimensional image information to achieve intelligent processing of the target two-dimensional image information. The volatile in-memory computing submodule 32 performs programmable digital signal processing on the target three-dimensional depth information to achieve reconstruction and depth discrimination of the target three-dimensional depth information. The non-volatile in-memory computing submodule 31 and the volatile in-memory computing submodule 32 are heterogeneous and parallel, and are programmed and controlled by a programmable microcontroller 50. This process mimics the human visual system's hybrid processing mechanism for two-dimensional image information and three-dimensional depth information.

[0034] In some exemplary embodiments, the signal shaping and encoding module 20 shapes and encodes the raw pulse imaging signal output by the single-photon array imaging sensor. Since the pulse imaging signal output by the single-photon array imaging sensor contains two-dimensional image information and three-dimensional depth information, the signal contains noise components, and the signal waveform is not a standard digital signal waveform. Directly inputting it into the processing module 30 for processing would result in unsatisfactory image processing effects. Therefore, processing is required in the signal shaping and encoding module 20, including signal shaping and encoding. Signal shaping is used to generate a clear and complete pulse imaging signal, and the shaped pulse imaging signal is input into the processing module 30. Signal encoding is used to perform row-column driven encoding on the two-dimensional image signal and the three-dimensional depth signal output by the single-photon array imaging sensor 10. The two-dimensional image signal is encoded into a signal format adapted to the non-volatile in-memory computing submodule 31, and the three-dimensional depth signal is encoded into a signal format adapted to the volatile in-memory computing submodule 32. The encoded signals are then input into the processing module 30.

[0035] The sensor-in-memory bionic visual imaging and recognition device 100 directly connects the imaging sensor 10 and the processing module 30 in the same device, and achieves the goal of low power consumption and low latency by mimicking the image acquisition and processing mode of the human visual system in terms of both image signal form and signal processing mode.

[0036] In terms of image signal format, the single-photon array imaging sensor can generate sparse pulse signals of the object under test, which simultaneously contain two-dimensional image information and three-dimensional depth information. The in-memory computing heterogeneous parallel processing module 30 can directly perform intelligent processing of the pulse signals output by the single-photon array imaging sensor using pulse convolutional neural networks, and simultaneously perform heterogeneous parallel processing of two-dimensional image information and three-dimensional depth information. This process mimics the physiological mechanism of human retina in perceiving and processing pulse image signals, as well as the physiological process of recognizing three-dimensional stereo images, reducing the amount of data transmitted between the imaging sensor 10 and the processing module 30, while also reducing the latency and energy consumption generated during data transmission.

[0037] In terms of signal processing, the processing module 30 consists of a non-volatile in-memory computing submodule 31 and a volatile in-memory computing submodule 32. The non-volatile in-memory computing submodule 31 is mainly used to process two-dimensional image information, while the volatile in-memory computing submodule 32 is mainly used to process three-dimensional depth information. The two submodules share a storage system 33 and are in a heterogeneous parallel relationship, controlled by a programmable microcontroller 50. This process mimics the human visual system's mechanism for processing a mixture of two-dimensional image information and three-dimensional depth information.

[0038] Figure 2 A schematic diagram of the structure of the processing module 30 according to an embodiment of the present disclosure is shown.

[0039] like Figure 2 As shown, the processing module 30 according to an embodiment of this disclosure includes a non-volatile in-memory computing submodule 31, a volatile in-memory computing submodule 32, and a storage system 33. The non-volatile in-memory computing submodule 31 is used to perform pulse convolutional neural network deep learning and processing on the target two-dimensional image information to achieve intelligent processing of the target two-dimensional image information and obtain the two-dimensional image recognition processing result in the analog domain. The volatile in-memory computing submodule 32 is used to perform programmable digital signal processing on the target three-dimensional depth information to achieve reconstruction and depth discrimination of the target three-dimensional depth information and obtain the three-dimensional image reconstruction and depth discrimination result in the digital domain.

[0040] For example, the non-volatile in-memory computing submodule 31 is composed of an array of resistive variable memory units 311. Each resistive variable memory unit 311 has non-volatile characteristics and in-memory computing characteristics, enabling it to non-volatilely store pulse convolutional neural network model parameters. Within the resistive variable memory unit 311, it performs multiplication and accumulation operations on the model parameters and input image information in the analog domain. This is used for deep learning and processing of pulse convolutional neural networks in the analog domain on pulsed two-dimensional image information, realizing intelligent image information processing and outputting the calculation results in the analog domain. Optionally, the resistive variable memory unit 311 adopts an xTyR structure, meaning that the resistive variable memory unit 311 is composed of x transistors and y resistive resistors, and each resistive variable memory unit 311 can store a signal with a width of m bits. Optionally, the input signal of the resistive variable memory unit 311 is a digital signal, and the output signal is a current-mode analog signal.

[0041] In this embodiment, the non-volatile nature of the resistive variable memory cell 311 enables the processing module 30 to retain neural network model parameters even after power failure, saving the delay time of input parameters after each power-on. Furthermore, the analog domain multiply-accumulate operations in the resistive variable memory cell 311 have a higher energy efficiency ratio compared to logical calculations in the digital domain.

[0042] Furthermore, the spiking convolutional neural network integrates the high recognition accuracy of traditional convolutional neural networks with the fully biomimetic characteristics of spiking neural networks, constructing a new visual system architecture by simulating the brain structure. Signal transmission between processing core units is accomplished through pulse signals; that is, the neural network image processing is performed through the transmission of pulse signals. The non-volatile in-memory computing submodule 31 uses resistive variable memory units 311 as the basic building block of the array. The interconnected processing core units, through their connections with surrounding units, form an efficient, scalable, and large-scale neural network.

[0043] For example, the volatile in-memory computing submodule 32 is composed of an array of static random access memory (SRAM) units 321. It can perform high-precision logical operations within the SRAM units 321, enabling programmable digital signal processing of 3D depth information to achieve 3D image reconstruction and depth discrimination, distinguishing between stereoscopic objects and planar photographs, and outputting the results. Optionally, the SRAM unit 321 is constructed from n transistors. The SRAM unit 321 is volatile, maintaining its stored data while the device is powered on, but losing the stored data upon power failure. Optionally, the input signal and output signal of the SRAM unit 321 are both digital signals.

[0044] In this embodiment, the non-volatile in-memory computing submodule 31 and the volatile in-memory computing submodule 32 share the storage system 33. The non-volatile in-memory computing submodule 31 stores the intermediate results of each convolutional layer calculation during the operation of the pulse convolutional neural network in the storage system 33. After each layer calculation is completed, the intermediate results can be overwritten. The volatile in-memory computing submodule 32 stores the intermediate results of logical calculations during 3D depth information image reconstruction and depth discrimination in the storage system 33. After each frame image calculation is completed, the intermediate results can be overwritten. The read and write operations of the storage system 33 are uniformly and synchronously controlled by the programmable microcontroller 50. The programmable microcontroller 50 can flexibly encode and implement different operations according to different application scenarios, and is controlled by operation instruction control signals and memory read and write control signals.

[0045] In this embodiment, both the resistive random access memory unit 311 and the static random access memory unit 321 have the characteristic of in-memory computing, which enables the processing module 30 to perform multiplication and accumulation operations at extremely high speed and extremely low power consumption, eliminating the latency and energy consumption of data transmission between traditional discrete memory and processor.

[0046] In some exemplary embodiments, the output module 40 is used to synchronize and output the recognition results of the processing module 30. Optionally, the recognition results include two-dimensional image recognition processing results in the analog domain and three-dimensional image reconstruction and depth discrimination results in the digital domain; the output module 40 converts the two-dimensional image recognition processing results in the analog domain into two-dimensional image recognition processing results in the digital domain; and the output module 40 integrates the two-dimensional image recognition processing results in the digital domain with the three-dimensional image reconstruction and depth discrimination results in the digital domain and outputs them synchronously. The output of the output module 40 is the output of the entire device.

[0047] In some exemplary embodiments, the programmable microcontroller 50 supports a RISC-V (Russibly Instruction Set Computer-V) architecture to provide synchronization control signals to the imaging sensor 10, signal shaping and encoding module 20, processing module 30, and output module 40. It programmably controls the heterogeneous parallel computing of the non-volatile in-memory computing submodule 31 and the volatile in-memory computing submodule 32 within the in-memory computing heterogeneous parallel processing module 30, and controls the synchronous and coordinated operation of the entire device. The RISC-V architecture features low power consumption, low cost, flexibility, scalability, and high security. It can be flexibly programmed and customized according to application requirements, has a streamlined instruction set and a compact kernel, and can achieve efficient instruction execution with extremely low power consumption.

[0048] In some exemplary embodiments, the programmable microcontroller 50 receives an input synchronization clock signal and provides synchronization control signals to the imaging sensor 10, the signal shaping and encoding module 20, the processing module 30, and the output module 40 through the synchronization clock signal.

[0049] In some exemplary embodiments, the programmable microcontroller 50 is bidirectionally connected to at least one of the following modules: the imaging sensor 10, the signal shaping and encoding module 20, the processing module 30, and the result output module 40, for transmitting control signals to the connection module and receiving feedback signals from the connection module.

[0050] In this embodiment, the characteristics of single-photon imaging and signal output are similar to the mechanism by which the human retina perceives images and outputs electrical pulse signals. The pulsed convolutional neural network integrates the characteristics of traditional convolutional neural networks and spiking neural networks, mimicking the way human visual information is transmitted and processed. Integrating the two functional modules into the same biomimetic visual imaging and recognition device 100 can naturally mimic the working mechanism of the human brain's visual system, and has the advantages of high performance and low power consumption.

[0051] The visual imaging and recognition device 100 provided in this embodiment can be used for imaging in low light environments. It has the function of fusing two-dimensional image information and three-dimensional depth information. It can quickly identify the target and reconstruct the depth information of the object under test with extremely low power consumption. It can distinguish whether the object under test is a three-dimensional object or a two-dimensional photograph. It can be applied to application scenarios such as intelligent security, face recognition, robot vision, and intelligent assisted driving.

[0052] The embodiments of this disclosure have now been described in detail with reference to the accompanying drawings.

[0053] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of this disclosure. It should be understood that the above are only specific embodiments of this disclosure and are not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A visual imaging and recognition device, characterized in that, include: Imaging sensors are used to acquire two-dimensional image information and three-dimensional depth information of a target object; The signal shaping and encoding module is used to perform signal shaping and encoding on the two-dimensional image information and the three-dimensional depth information to obtain the target two-dimensional image information and the target three-dimensional depth information. The processing module is used to process and recognize the target's two-dimensional image information and the target's three-dimensional depth information to generate recognition results; The output module is used to perform signal synchronization buffering and output of the recognition results; as well as A programmable microcontroller is used to provide synchronous control signals to the imaging sensor, the signal shaping and encoding module, the processing module and the output module, so as to control the device to operate synchronously and in a coordinated manner; The processing module includes: A non-volatile in-memory computing submodule is used to perform pulsed convolutional neural network deep learning and processing on the target two-dimensional image information to achieve intelligent processing of the target two-dimensional image information and obtain the two-dimensional image recognition processing result in the analog domain; and A volatile in-memory computing submodule is used to perform programmable digital signal processing on the target's three-dimensional depth information to reconstruct and determine the target's three-dimensional depth information, thereby obtaining digital domain three-dimensional image reconstruction and depth determination results. The non-volatile in-memory computing submodule and the volatile in-memory computing submodule are heterogeneous and parallel, and are programmed and controlled by the programmable microcontroller to perform mixed processing of two-dimensional image information and three-dimensional depth information. The recognition results include the two-dimensional image recognition processing results in the analog domain and the three-dimensional image reconstruction and depth discrimination results in the digital domain; The output module is used to convert the two-dimensional image recognition processing result in the analog domain into a two-dimensional image recognition processing result in the digital domain; and The output module is used to integrate the two-dimensional image recognition processing results of the digital domain with the three-dimensional image reconstruction and depth discrimination results of the digital domain and output them synchronously.

2. The apparatus according to claim 1, characterized in that, The imaging sensor includes a single-photon array imaging sensor.

3. The apparatus according to claim 1 or 2, characterized in that, The two-dimensional image information includes pulsed two-dimensional image information, wherein the pulse frequency of the pulsed two-dimensional image information is positively correlated with the intensity of the light incident signal of the target object; and The method for measuring the three-dimensional depth information includes the time-of-flight method.

4. The apparatus according to claim 1, characterized in that, The processing module further includes: Storage system The non-volatile in-memory computing submodule and the volatile in-memory computing submodule share the storage system.

5. The apparatus according to claim 1, characterized in that, The non-volatile in-memory computing submodule includes a resistive variable memory cell array, wherein the resistive variable memory cells have non-volatile characteristics and in-memory computing characteristics; and / or The resistive switching memory cell array includes transistors and resistive switching resistors; and / or The input signals of the resistive variable memory cell include digital signals, and the output signals include analog signals in the current mode.

6. The apparatus according to claim 1, characterized in that, The volatile in-memory computing submodule includes a static random access memory (SRAM) cell array, wherein the SRAM cells are capable of performing logical operations within the SRAM cells; and / or The static random access memory unit includes transistors; and / or The input signals of the static random access memory unit include digital signals, and the output signals include digital signals.

7. The apparatus according to claim 1, characterized in that, The programmable microcontroller receives an input synchronization clock signal and provides synchronization control signals to the imaging sensor, the signal shaping and encoding module, the processing module, and the output module through the synchronization clock signal; and / or The programmable microcontroller supports the RISC-V architecture.

8. The apparatus according to claim 1, characterized in that, The programmable microcontroller is bidirectionally connected to at least one of the following modules: the imaging sensor, the signal shaping and encoding module, the processing module, and the output module. It is used to transmit control signals to the connection module and receive feedback signals from the connection module.

Images