Programmable visible light infrared vision chip

Abstract

The invention provides a programmable visible light infrared visual chip, and relates to the technical field of visible light and infrared visual imaging and visual information processing. Comprising a visible light-infrared detector array; a pixel-level readout circuit array; a sensor digital control circuit; a visible light-infrared image preprocessor; a visible light-infrared image heterogeneous processor; a data output interface circuit; and the metal column array is used for carrying out three-dimensional stacking interconnection on the visible light-infrared detector array, the pixel-level reading circuit array, the visible light-infrared image preprocessor and the visible light-infrared image heterogeneous processor. According to the method, through three-dimensional heterogeneous integration of the visible light-infrared detector array and the pixel-level reading circuit array, a large-scale visible light-infrared visual imaging task is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of visible light and infrared vision imaging and vision information processing, and particularly to a programmable visible light infrared vision chip. Background Art

[0002] Currently, vision information processing technology is developing at an unprecedented speed, and its core development needs focus on high-speed imaging, extracting the required target information from vision information with high precision and low latency. To meet these urgent application needs, sensor-computation integrated vision chips have become the focus of industry research.

[0003] Currently, most vision chips are based on imaging arrays in the visible light band. However, the vision information in the visible light band cannot meet the application requirements in specific scenarios. The vision information in the infrared band includes the thermal imaging information of the target and the transmission / reflection information different from visible light, and has broad application prospects in fields such as industrial and agricultural detection and embodied intelligence. However, the current visible light infrared vision chips still have problems such as low imaging speed, dynamic blind pixels, and non-uniformity correction, and cannot perform intelligent analysis for different processing tasks, lacking flexibility, resulting in the inability to achieve high-integration, high-speed, and high-energy efficiency exploration-computation integrated infrared vision intelligent processing. Summary of the Invention

[0004] In view of the above problems, an embodiment of the present invention provides a programmable visible light infrared vision chip.

[0005] One aspect of the present invention provides a programmable visible light infrared vision chip, including: a visible light-infrared detector array for detecting optical signals in the visible light infrared band and converting the optical signals into current signals; a pixel-level readout circuit array for performing image conversion on the current signals according to imaging parameters to obtain image data; a sensor digital control circuit for adjusting the imaging parameters according to control information and transmitting the image data; a visible light-infrared image pre-processor for performing correction processing on the image data to obtain corrected data; a visible light-infrared image heterogeneous processor for issuing control information and analyzing the corrected data to obtain an analysis result; a data output interface circuit for receiving the analysis result and outputting it; and a metal pillar array for three-dimensional stacked interconnection of the visible light-infrared detector array with the pixel-level readout circuit array, the visible light-infrared image pre-processor, and the visible light-infrared image heterogeneous processor.

[0006] According to an embodiment of the present invention, the programmable visible light-infrared vision chip further includes a data input interface circuit, a test module, and a phase-locked loop; the data input interface circuit includes a test control signal interface circuit for acquiring a test control signal and transmitting the test control signal to the test module; the test module is used to perform a function test according to the test control signal; the phase-locked loop is used to configure the main frequency clock for the chip to operate.

[0007] According to an embodiment of the present invention, the data input interface circuit is further used to receive input parameters, and the input parameters include chip parameters and initialization signals; wherein, the input parameters are used to initialize the visible light-infrared detector array, the pixel-level readout circuit array, the sensor digital control circuit, the visible light-infrared image pre-processor, and the visible light-infrared image heterogeneous processor respectively.

[0008] According to an embodiment of the present invention, the data input interface circuit further includes: a high-speed serializer / deserializer input interface circuit for receiving data input from outside the chip; a parallel input interface circuit and a serial input interface circuit for receiving chip parameters; and an initialization signal interface circuit for receiving initialization signals.

[0009] According to an embodiment of the present invention, the visible light-infrared detector array includes M×N visible light-infrared detector units for receiving optical signals and converting the optical signals into current signals, where M represents the number of rows, N represents the number of columns, and M and N are positive integers respectively.

[0010] According to an embodiment of the present invention, the pixel-level readout circuit array includes: M×N pixel-level readout circuit units for accumulating current signals and converting them into image data.

[0011] According to an embodiment of the present invention, the metal column array includes: M×N metal columns, and each metal column is connected to a visible light-infrared detector unit and a pixel-level readout circuit unit, and the metal columns are used to transfer current signals.

[0012] According to an embodiment of the present invention, the sensor digital control circuit includes: a message parsing module for parsing control information to obtain control instructions; a pulse generation module for sending a plurality of pulse signals to the pixel-level readout circuit array according to the control instructions, and the pulse signals are used to adjust imaging parameters; a result sending module for transmitting image data to the visible light-infrared image pre-processor.

[0013] According to an embodiment of the present invention, the visible light-infrared image pre-processor includes: a multiple correction module for performing blind pixel correction and non-uniformity correction processing on the image data to obtain second image data; a histogram stretching module for adaptively stretching the second image data to obtain third image data; an image cropping module for cropping the third image data to obtain corrected data.

[0014] According to an embodiment of the present invention, the histogram stretching module includes: a histogram statistics circuit for statistically analyzing the gray levels of the second image data to obtain a statistical result; a threshold discrimination circuit for determining a mapping threshold according to the statistical result; and a histogram mapping circuit for stretching the second image data according to the statistical result and the mapping threshold to obtain third image data.

[0015] According to an embodiment of the present invention, the image cropping module includes: a sequential storage circuit for sequentially storing the third image data; and a memory access type cropping circuit for cropping the third image data through a customized memory access method to obtain corrected data.

[0016] According to an embodiment of the present invention, the visible light-infrared image heterogeneous processor includes: a microcontroller core for sending control information and parsing small-scale serial corrected data; a vector processor core for parsing large-scale parallel corrected data; a neural network processor core for parsing neural network-related corrected data; a data memory for storing the corrected data; and a processor data memory access circuit for realizing the input and output of the corrected data.

[0017] According to an embodiment of the present invention, the microcontroller core includes: a microcontroller instruction state machine for controlling the microcontroller core; a message encoding / decoding transceiver module for receiving messages from each module and sending control information; and a microcontroller processing circuit for parsing small-scale serial corrected data.

[0018] According to an embodiment of the present invention, the message encoding / decoding transceiver module includes: a processor state monitoring module for monitoring the internal environment of the chip to provide decision-making information for the microcontroller core; a peripheral message receiving module for receiving messages from each module; and a peripheral message sending module for sending control information.

[0019] According to an embodiment of the present invention, the vector processor core includes: a vector instruction state machine for controlling the vector processor core; a global processing circuit for parsing single-thread corrected data; and a vector processing circuit for parsing large-scale parallel corrected data. According to an embodiment of the present invention, the global processing circuit includes: a global register for storing single-thread corrected data; and a global arithmetic unit for performing related operations on the single-thread corrected data.

[0020] According to an embodiment of the present invention, the vector processing circuit includes: a vector register for storing large-scale corrected data; and a vector arithmetic unit for performing related operations on the large-scale corrected data.

[0021] According to an embodiment of the present invention, a neural network processor core includes: a neural network instruction state machine for controlling the behavior of the neural network processor core; a neural network data access circuit for implementing the input and output of calibration data; a weight storage circuit for storing and receiving / sending weight data required for neural network calculations; a feature map storage circuit for storing and receiving / sending feature map data required for neural network calculations; a neural network processing array for performing convolution calculations on calibration data related to the neural network; an accumulator for performing accumulation operations on calibration data related to the neural network; and a neural network post-processing circuit for performing post-processing on calibration data related to the neural network.

[0022] According to an embodiment of the present invention, the neural network data access circuit includes: a weight storage input transporter for transferring weight data from a data memory to the weight storage circuit; a feature map storage input transporter for transferring feature map data from the data memory to the feature map storage circuit; a neural network post-processing output transporter for transferring the result of the operation completed by the neural network post-processing circuit back to the data memory.

[0023] According to an embodiment of the present invention, the weight storage circuit includes: a weight memory for storing weight data required for neural network calculations; a weight buffer output module for buffering and outputting the data in the weight memory to the neural network processing array.

[0024] According to an embodiment of the present invention, the feature map storage circuit includes: a feature map memory for storing feature map data required for neural network calculations; a feature map buffer output module for buffering and outputting the data in the feature map memory to the neural network processing array.

[0025] According to an embodiment of the present invention, the neural network processing array includes: Q×Q neural network processing units for completing a single operation of neural network convolution operations, where Q represents the number of rows and columns, and Q is a positive integer; a programmable connector for dynamically combining the neural network processing units to complete the convolution operations of the neural network.

[0026] According to an embodiment of the present invention, a neural network processing unit includes: a weight register for storing weight data to be operated; a feature map register for storing feature map data to be operated; a calculation unit for performing calculations based on the weight data and the feature map data; and a result register for storing the result data after the operation of the calculation unit is completed.

[0027] According to an embodiment of the present invention, the data output interface circuit 20 includes: a high-speed serializer / deserializer output interface circuit and a parallel output interface circuit for receiving and outputting the parsing result; a message output interface circuit for sending custom messages.

[0028] The method of the present invention realizes large-scale visible-infrared visual imaging tasks through the three-dimensional heterogeneous integration of a visible-infrared detector array and a pixel-level readout circuit array. It also compensates for the inherent blind pixels and non-uniformity defects of the visible-infrared detector array through a visible-infrared image preprocessor, and realizes the parsing of different processing tasks through a visible-infrared image heterogeneous processor, with high flexibility, solving the problems of blind pixels and non-uniformity existing in current infrared vision chips, as well as the lack of flexibility in parsing for different processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Through the following description of the embodiments of the present invention with reference to the drawings, the above content and other objects, features, and advantages of the present invention will become clearer. In the drawings:

[0030] Figure 1 Schematically shows one of the structural diagrams of a programmable visible-infrared vision chip according to an embodiment of the present invention;

[0031] Figure 2 Schematically shows another structural diagram of a programmable visible-infrared vision chip according to an embodiment of the present invention;

[0032] Figure 3 Schematically shows the structural diagram of the sensing and readout part according to an embodiment of the present invention;

[0033] Figure 4 Schematically shows the structural diagram of a visible-infrared image preprocessor according to an embodiment of the present invention;

[0034] Figure 5 Schematically shows the structural diagram of a visible-infrared image heterogeneous processor according to an embodiment of the present invention;

[0035] Figure 6 Schematically shows the data flow diagram of a programmable visible-infrared vision chip according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the following further describes the present invention in detail with reference to specific embodiments and the accompanying drawings.

[0037] It should be noted that in the drawings or the description of the specification, similar or identical parts are all denoted by the same reference numerals. The technical features in each of the embodiments exemplified in the specification can be freely combined to form a new solution on the premise of no conflict. Moreover, in the drawings, the shape or thickness of the embodiments can be enlarged, and can be simplified or conveniently marked. Furthermore, the elements or implementation manners not shown or described in the drawings are in the forms known to those of ordinary skill in the art. In addition, although this document may provide examples including parameters with specific values, it should be understood that the parameters do not necessarily exactly equal the corresponding values, but may approximate the corresponding values within an acceptable error tolerance or design constraint.

[0038] Unless there are technical obstacles or contradictions, the above various embodiments of the present invention can be freely combined to form additional embodiments, and these additional embodiments are all within the protection scope of the present invention.

[0039] Although the present invention has been described in conjunction with the drawings, the embodiments disclosed in the drawings are intended to exemplarily illustrate the preferred embodiments of the present invention and should not be construed as a limitation on the present invention. The dimensional ratios in the drawings are merely illustrative and should not be construed as a limitation on the present invention.

[0040] Although some embodiments of the general concept of the present invention have been shown and described, those of ordinary skill in the art will understand that changes can be made to these embodiments without departing from the principles and spirit of this general disclosure.

[0041] Figure 1 One of the structural diagrams of a programmable visible light infrared vision chip according to an embodiment of the present invention is schematically shown.

[0042] As Figure 1 shown, an embodiment of the present invention provides a programmable visible light infrared vision chip, including: a visible light-infrared detector array 30 for detecting light signals in the visible light infrared band and converting the light signals into current signals; a pixel-level readout circuit array 40 for performing image conversion on the current signals according to imaging parameters to obtain image data; a sensor digital control circuit 60 for adjusting the imaging parameters according to control information and transmitting the image data; a visible light-infrared image pre-processor 70 for performing correction processing on the image data to obtain corrected data; a visible light-infrared image heterogeneous processor 80 for sending control information and parsing the corrected data to obtain a parsing result; a data output interface circuit 20 for receiving the parsing result and outputting it; and a metal pillar array 50 for three-dimensionally stacking and interconnecting the visible light-infrared detector array 30 with the pixel-level readout circuit array 40, the visible light-infrared image pre-processor 70, and the visible light-infrared image heterogeneous processor 80.

[0043] The method provided by the embodiments of the present invention realizes large-scale visible light-infrared vision imaging tasks through the three-dimensional heterogeneous integration of a visible light-infrared detector array and a pixel-level readout circuit array; through a visible light-infrared image preprocessor, it realizes the compensation for the inherent blind pixels and non-uniformity defects of the visible light-infrared detector array; through a visible light-infrared image heterogeneous processor, it realizes the parsing of different processing tasks, with high flexibility. Therefore, this method can solve the problems of blind pixels and non-uniformity existing in current infrared vision chips, as well as the lack of flexibility in parsing different processing tasks.

[0044] Figure 2 Schematically shows the second structural diagram of a programmable visible light-infrared vision chip according to an embodiment of the present invention.

[0045] As Figure 2 shown, the programmable visible light-infrared vision chip of this embodiment further includes a data input interface circuit 10, a test module 100, and a phase-locked loop (101); the data input interface circuit 10 includes a test control signal interface circuit 140, which is used to obtain a test control signal and transmit the test control signal to the test module 100; the test module 100 is used to perform a function test according to the test control signal; the phase-locked loop 101 is used to configure the main clock frequency for the chip to operate.

[0046] In some embodiments, when the chip is in the test mode, the test module 100 sends the information of the key registers of the chip outside the chip, which is used to quickly detect the effectiveness of the chip and locate the failed module.

[0047] In some embodiments, the data input interface circuit 10 is further used to receive input parameters, and the input parameters include chip parameters and initialization signals; among them, the input parameters are used to initialize the visible light-infrared detector array 30, the pixel-level readout circuit array 40, the sensor digital control circuit 60, the visible light-infrared image preprocessor 70, and the visible light-infrared image heterogeneous processor 80 respectively.

[0048] Please continue to refer to Figure 2 , the data input interface circuit 10 further includes: a high-speed serializer / deserializer input interface circuit 110, which is used to receive the data input outside the chip; a parallel input interface circuit 120 and a serial input interface circuit 130, which are used to receive chip parameters; and an initialization signal interface circuit 150, which is used to receive initialization signals.

[0049] Figure 3 Schematically shows the structural diagram of the sensing and readout part according to an embodiment of the present invention.

[0050] As Figure 3As shown, the visible light-infrared detector array 30 of this embodiment includes M×N visible light-infrared detector units 310, which are used to receive optical signals and convert the optical signals into current signals. Here, M represents the number of rows, N represents the number of columns, and M and N are positive integers respectively.

[0051] Please continue to refer to Figure 3 , the pixel-level readout circuit array 40 includes: M×N pixel-level readout circuit units 410, which are used to accumulate current signals and convert them into image data.

[0052] Among them, the pixel-level readout circuit unit 410 can accumulate the current signals generated by the detector units and convert them into pixel values in the digital circuit signal format of P bits for subsequent digital circuit processing, where P is a positive integer.

[0053] Among them, the metal pillar array 50 includes: M×N metal pillars, and each metal pillar is connected to a visible light-infrared detector unit 310 and a pixel-level readout circuit unit 410. The metal pillars are used to transmit current signals.

[0054] In some embodiments, the programmable visible light-infrared vision chip may include two sub-chips. The upper layer is a sub-chip composed of the visible light-infrared detector array 30, and the lower layer is a second sub-chip composed of the pixel-level readout circuit array 40 and the remaining circuits.

[0055] Please continue to refer to Figure 3 , the sensor digital control circuit 60 includes: a message parsing module 610, which is used to parse control information to obtain control instructions; a pulse generation module 620, which is used to send multiple pulse signals to the pixel-level readout circuit array 40 according to the control instructions, and the pulse signals are used to adjust imaging parameters; a result sending module 630, which is used to transmit the image data to the visible light-infrared image pre-processor 70.

[0056] Figure 4 Schematically shows the structural diagram of the visible light-infrared image pre-processor according to an embodiment of the present invention.

[0057] As Figure 4 shown, the visible light-infrared image pre-processor 70 includes: a multiple correction module 710, which is used to perform blind pixel correction and non-uniformity correction processing on the image data to obtain second image data; a histogram stretching module 720, which is used to perform adaptive stretching on the second image data to obtain third image data; an image cropping module 730, which is used to crop the third image data to obtain corrected data.

[0058] Among them, the histogram stretching module 720 includes: a histogram statistics circuit 7210 for statistically analyzing the gray levels of the second image data to obtain a statistical result; a threshold discrimination circuit 7220 for determining a mapping threshold according to the statistical result; and a histogram mapping circuit 7230 for stretching the second image data according to the statistical result and the mapping threshold to obtain third image data.

[0059] In some embodiments, the image cropping module 730 includes: an ordered storage circuit 7310 for storing the third image data in order; and a memory access-based cropping circuit 7320 for cropping the third image data through a custom memory access method to obtain corrected data.

[0060] Figure 5 Schematically shows a structural diagram of a visible light-infrared image heterogeneous processor according to an embodiment of the present invention.

[0061] As Figure 5 shown, the visible light-infrared image heterogeneous processor 80 includes: a microcontroller core 810 for sending control information and parsing small-scale serial corrected data; a vector processor core 820 for parsing large-scale parallel corrected data; a neural network processor core 830 for parsing neural network-related corrected data; a data memory 840 for storing corrected data; and a processor data memory access circuit 850 for implementing the input and output of corrected data.

[0062] Among them, the microcontroller core 810 includes: a microcontroller instruction state machine 8110 for controlling the microcontroller core 810; a message encoding / decoding transceiver module 8120 for receiving messages from each module and sending control information; and a microcontroller processing circuit 8130 for parsing small-scale serial corrected data.

[0063] In some embodiments, the message encoding / decoding transceiver module 8120 includes: a processor state monitoring module 81201 for monitoring the internal environment of the chip to provide decision-making information to the microcontroller core 810; a peripheral message receiving module 81202 for receiving messages from each module; and a peripheral message sending module 81203 for sending control information.

[0064] In some embodiments, the vector processor core 820 includes: a vector instruction state machine 8210 for controlling the vector processor core 820; a global processing circuit 8220 for parsing single-thread corrected data; and a vector processing circuit 8230 for parsing large-scale parallel corrected data.

[0065] In some embodiments, the global processing circuit 8220 includes: a global register 82201 for storing calibration data of a single thread; and a global arithmetic unit 82202 for performing operations related to the calibration data of the single thread.

[0066] In some embodiments, the vector processing circuit 8230 includes: a vector register 82301 for storing large-scale calibration data; and a vector arithmetic unit 82302 for performing operations related to the large-scale calibration data.

[0067] In some embodiments, the neural network processor core 830 includes: a neural network instruction state machine 8310 for controlling the behavior of the neural network processor core 830; a neural network data access circuit 8320 for implementing the input and output of calibration data; a weight storage circuit 8330 for storing and transmitting weight data required for neural network calculations; a feature map storage circuit 8340 for storing and transmitting feature map data required for neural network calculations; a neural network processing array 8350 for performing convolution calculations on calibration data related to the neural network; an accumulator 8360 for performing accumulation operations on calibration data related to the neural network; and a neural network post-processing circuit 8370 for performing post-processing on calibration data related to the neural network.

[0068] In some embodiments, the neural network data access circuit 8320 includes: a weight storage input transporter for transferring weight data from the data memory 840 to the weight storage circuit 8330; a feature map storage input transporter for transferring feature map data from the data memory 840 to the feature map storage circuit 8340; and a neural network post-processing output transporter for transferring the result of the operation of the neural network post-processing circuit 8370 back to the data memory 840.

[0069] In some embodiments, the weight storage circuit 8330 includes: a weight memory for storing weight data required for neural network calculations; and a weight buffer output module for caching and outputting the data in the weight memory to the neural network processing array.

[0070] In some embodiments, the feature map storage circuit 8340 includes: a feature map memory for storing feature map data required for neural network calculations; and a feature map buffer output module for caching and outputting the data in the feature map memory to the neural network processing array 8350.

[0071] In some embodiments, the neural network processing array 8350 includes: Q×Q neural network processing units 83501 for completing a single operation of neural network convolution operation, where Q represents the number of rows and columns, and Q is a positive integer; a programmable connector 83502 for dynamically combining the neural network processing units to complete the convolution operation of the neural network.

[0072] In some embodiments, the neural network processing unit includes: a weight register for storing weight data to be operated; a feature map register for storing feature map data to be operated; a calculation unit for calculating according to the weight data and the feature map data; and a result register for storing the result data after the operation of the calculation unit is completed.

[0073] In some embodiments, the data output interface circuit 20 includes: a high-speed serializer / deserializer output interface circuit 210 and a parallel output interface circuit 220 for receiving the parsing result and outputting it; a message output interface circuit 230 for sending custom messages.

[0074] Figure 6 Schematically shows the data flow diagram of the programmable visible light infrared vision chip according to an embodiment of the present invention.

[0075] As Figure 6 shown, the data flow of the embodiment of the present invention includes steps S610 to S650.

[0076] Step S610, globally reset the programmable visible light infrared vision chip.

[0077] Step S620, load infrared heterogeneous image processor instructions from the initialization signal interface.

[0078] Step S640, load algorithm parameters from the serial input interface.

[0079] Step S650, the microcontroller performs a parsing operation.

[0080] Step S660, the operation of the programmable visible light infrared vision chip ends.

[0081] Further, the above step S640 may further include:

[0082] Sub-step S6401, interact with the outside through the input interface.

[0083] Sub-step S6402, perform a parsing operation through the vector processor core.

[0084] Sub-step S6403, perform a parsing operation through the neural network processor core.

[0085] After global reset, the programmable visible light and infrared vision chip according to the embodiment of the present invention loads visible light-infrared heterogeneous image processor instructions from the initialization signal interface circuit 150, and then loads algorithm parameters from the serial input interface circuit 130. After completion, it starts to execute instructions by the microcontroller. During the operation of the microcontroller, it may call the remaining parts of the chip to complete the following three operations: interact with the outside through the input / output interface circuit, execute instructions by the vector processor core, and execute instructions by the neural network processor core. After the operation of the microcontroller is completed, the operation of the chip ends.

[0086] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of the present invention.

[0087] It should also be noted that the directional terms mentioned in the embodiments, such as "up", "down", "front", "back", "left", "right", etc., are only references to the directions in the drawings and are not used to limit the scope of the present invention. Throughout the drawings, the same elements are represented by the same or similar reference numerals. When it may cause confusion in the understanding of the present invention, the conventional structures or configurations will be omitted. And the shapes, sizes, and positional relationships of the components in the drawings do not reflect the actual sizes, ratios, and actual positional relationships.

[0088] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined. Regarding the term "comprising" used in the specification, the coverage of this word is similar to the term "including", as explained when "including," is used as a connecting word.

[0089] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A programmable visible light and infrared vision chip, characterized in that Comprising: A visible light-infrared detector array (30) for detecting optical signals in the visible light-infrared band and converting the optical signals into current signals; An element-level readout circuit array (40) for performing image conversion on the current signals according to imaging parameters to obtain image data; A sensor digital control circuit (60) for adjusting the imaging parameters according to control information and transmitting the image data; A visible light-infrared image preprocessor (70) for performing correction processing on the image data to obtain corrected data; A visible light-infrared image heterogeneous processor (80) for issuing control information and parsing the corrected data to obtain a parsing result; A data output interface circuit (20) for receiving the parsing result and outputting it; A metal pillar array (50) for three-dimensionally stacking and interconnecting the visible light-infrared detector array (30) with the element-level readout circuit array (40), the visible light-infrared image preprocessor (70), and the visible light-infrared image heterogeneous processor (80).

2. The programmable visible light and infrared vision chip according to claim 1, characterized in that It further includes a data input interface circuit (10), a test module (100), and a phase-locked loop (101); The data input interface circuit (10) includes a test control signal interface circuit (140) for acquiring a test control signal and transmitting the test control signal to the test module (100); The test module (100) is used for performing a function test according to the test control signal; The phase-locked loop (101) is used for configuring the main clock frequency for the chip to operate.

3. The programmable visible light infrared vision chip according to claim 2, characterized in that, The data input interface circuit (10) is further used for receiving input parameters, and the input parameters include chip parameters and initialization signals; Wherein, the input parameters are used for respectively initializing the visible light-infrared detector array (30), the element-level readout circuit array (40), the sensor digital control circuit (60), the visible light-infrared image preprocessor (70), and the visible light-infrared image heterogeneous processor (80).

4. The programmable visible light and infrared vision chip according to claim 3, wherein, The data input interface circuit (10) further includes: A high-speed serializer / deserializer input interface circuit (110) for receiving data input outside the chip; A parallel input interface circuit (120) and a serial input interface circuit (130) for receiving the chip parameters; And an initialization signal interface circuit (150) for receiving the initialization signal.

5. The programmable visible light and infrared vision chip according to claim 1, characterized in that, The visible light-infrared detector array (30) includes: M×N visible light-infrared detector units (310) for receiving the optical signals and converting the optical signals into the current signals, where M represents the number of rows, N represents the number of columns, and M and N are positive integers respectively.

6. The programmable visible light infrared vision chip according to claim 5, wherein, The element-level readout circuit array (40) includes: M×N element-level readout circuit units (410) for accumulating the current signals and converting them into the image data.

7. The programmable visible light infrared vision chip according to claim 6, wherein The metal pillar array (50) includes: M×N metal columns, each of the metal columns connecting one of the visible light-infrared detector units (310) and one of the pixel-level readout circuit units (410), the metal columns being used to transfer the current signal.

8. The programmable visible light infrared vision chip according to claim 1, wherein The sensor digital control circuit (60) includes: A message parsing module (610) for parsing the control information to obtain a control instruction; A pulse generation module (620) for sending a plurality of pulse signals to the pixel-level readout circuit array (40) according to the control instruction, the pulse signals being used to adjust the imaging parameters; A result sending module (630) for transmitting the image data to the visible light-infrared image pre-processor (70).

9. The programmable visible light and infrared vision chip according to claim 1, wherein The visible light-infrared image pre-processor (70) includes: A multiple correction module (710) for performing blind pixel correction and non-uniformity correction processing on the image data to obtain second image data; A histogram stretching module (720) for adaptively stretching the second image data to obtain third image data; An image cropping module (730) for cropping the third image data to obtain the corrected data.

10. The programmable visible light infrared vision chip according to claim 9, wherein The histogram stretching module (720) includes: A histogram statistics circuit (7210) for statistically analyzing the gray levels of the second image data to obtain a statistical result; A threshold discrimination circuit (7220) for determining a mapping threshold according to the statistical result; A histogram mapping circuit (7230) for stretching the second image data according to the statistical result and the mapping threshold to obtain third image data.

11. The programmable visible light and infrared vision chip according to claim 9, wherein The image cropping module (730) includes: An ordered storage circuit (7310) for storing the third image data in order; An access-mode cropping circuit (7320) for cropping the third image data by a custom access mode to obtain the corrected data.

12. The programmable visible light infrared vision chip according to claim 1, wherein, The visible light-infrared image heterogeneous processor (80) includes: A microcontroller core (810) for sending the control information and parsing the small-scale serial corrected data; A vector processor core (820) for parsing the large-scale parallel corrected data; A neural network processor core (830) for parsing the corrected data related to the neural network; A data memory (840) for storing the corrected data; A processor data access circuit (850) for implementing the input and output of the corrected data.

13. The programmable visible light infrared vision chip according to claim 12, wherein, The microcontroller core (810) includes: A microcontroller instruction state machine (8110) for controlling the microcontroller core (810); A message encoding / decoding transceiver module (8120) for receiving messages from each module and sending the control information; A microcontroller processing circuit (8130) for parsing the small-scale serial corrected data.

14. The programmable visible light and infrared vision chip according to claim 13, characterized in that, The message encoding / decoding transceiver module (8120) includes: A processor state monitoring module (81201) for monitoring the internal environment of the chip to provide decision-making information for the microcontroller core (810); The peripheral message receiving module (81202) is used to receive messages from each module; The peripheral message sending module (81203) is used to send the control information.

15. The programmable visible light infrared vision chip according to claim 12, characterized in that, The vector processor core (820) includes: A vector instruction state machine (8210) for controlling the vector processor core (820); A global processing circuit (8220) for parsing the single-threaded correction data; A vector processing circuit (8230) for parsing the massively parallel correction data.

16. The programmable visible light and infrared vision chip according to claim 15, wherein The global processing circuit (8220) includes: A global register (82201) for storing the single-threaded correction data; A global arithmetic unit (82202) for performing related operations on the single-threaded correction data.

17. The programmable visible light infrared vision chip according to claim 15, characterized in that, The vector processing circuit (8230) includes: A vector register (82301) for storing the massively parallel correction data; A vector arithmetic unit (82302) for performing related operations on the massively parallel correction data.

18. The programmable visible light and infrared vision chip according to claim 12, characterized in that The neural network processor core (830) includes: A neural network instruction state machine (8310) for controlling the behavior of the neural network processor core (830); A neural network data access circuit (8320) for implementing the input and output of the correction data; A weight storage circuit (8330) for storing and transmitting the weight data required for neural network calculations; A feature map storage circuit (8340) for storing and transmitting the feature map data required for neural network calculations; A neural network processing array (8350) for performing convolution calculations on the neural network-related correction data; An accumulator (8360) for performing accumulation operations on the neural network-related correction data; And a neural network post-processing circuit (8370) for post-processing the neural network-related correction data.

19. The programmable visible light and infrared vision chip according to claim 18, characterized in that The neural network data access circuit (8320) includes: A weight storage input transporter for transferring weight data from the data memory (840) to the weight storage circuit (8330); A feature map storage input transporter for transferring feature map data from the data memory (840) to the feature map storage circuit (8340); A neural network post-processing output transporter for transferring the result of the neural network post-processing circuit (8370) after the operation back to the data memory (840).

20. The programmable visible light infrared vision chip according to claim 18, wherein The weight storage circuit (8330) includes: A weight memory for storing the weight data required for neural network calculations; A weight buffer output module for buffering and outputting the data in the weight memory to the neural network processing array.

21. The programmable visible light infrared vision chip according to claim 18, characterized in that, The feature map storage circuit (8340) includes: A feature map memory for storing the feature map data required for neural network calculations; A feature map buffer output module for buffering and outputting the data in the feature map memory to the neural network processing array (8350).

22. The programmable visible light and infrared vision chip according to claim 18, wherein The neural network processing array (8350) includes: Q × Q neural network processing units (83501) for performing a single operation of neural network convolution operation, where Q represents the number of rows and columns, and Q is a positive integer; A programmable connector (83502) for dynamically combining neural network processing units to complete the convolution operation of the neural network.

23. The programmable visible light infrared vision chip according to claim 22, characterized in that, The neural network processing unit includes: A weight register for storing weight data to be operated; A feature map register for storing feature map data to be operated; A calculation unit for calculating according to the weight data and the feature map data; A result register for storing the result data after the calculation unit completes the operation.

24. The programmable visible light and infrared vision chip according to claim 1, characterized in that, The data output interface circuit (20) includes: A high-speed serializer / deserializer output interface circuit (210) and a parallel output interface circuit (220) for receiving the parsing result and outputting it; A message output interface circuit (230) for sending custom messages.

Images