Programmable visible light infrared vision chip

Abstract

The present invention provides a programmable visible-light infrared vision chip, which relates to the fields of visible-light and infrared visual imaging and visual information processing. The chip comprises a visible-light infrared detector array; a pixel-level readout circuit array; a sensor digital control circuit; a visible-light infrared image preprocessor; a visible-light infrared image heterogeneous processor; a data output interface circuit; and a metal pillar array for three-dimensionally stacking and interconnecting the visible-light infrared detector array, the pixel-level readout circuit array, the visible-light infrared image preprocessor, and the visible-light infrared image heterogeneous processor. The present invention's method achieves large-scale visible-light infrared visual imaging tasks through the three-dimensional heterogeneous integration of the visible-light infrared detector array and the pixel-level readout circuit array.

Description

Technical Field

[0001] The present invention relates to the technical field of visible light and infrared visual imaging and visual information processing, and in particular to a programmable visible light infrared vision chip. Background Art

[0002] Visual information processing technology is currently developing at an unprecedented pace, with core development requirements focused on high-speed imaging and extracting target information from visual data with high precision and low latency. To meet these pressing application demands, integrated sensing and computing vision chips have become a research focus in the industry.

[0003] Currently, most vision chips are based on imaging arrays in the visible light band. However, the visual information in the visible light band cannot meet the application requirements of specific scenarios. Infrared band visual information, including thermal imaging information of the target and transmission / reflection information that is different from visible light, has broad application prospects in industrial and agricultural inspection, embodied intelligence, and other fields. However, current visible light infrared vision chips still have problems such as low imaging speed, dynamic blind pixels, and non-uniformity correction. They are also unable to intelligently analyze different processing tasks and lack flexibility, making it impossible to achieve highly integrated, high-speed, and energy-efficient infrared vision intelligent processing that integrates detection and computing. Summary of the Invention

[0004] In view of the above problems, an embodiment of the present invention provides a programmable visible light infrared vision chip.

[0005] One aspect of the present invention provides a programmable visible light infrared vision chip, comprising: a visible light-infrared detector array for detecting light signals in the visible light infrared band and converting the light signals into current signals; a pixel-level readout circuit array for performing image conversion on the current signals according to imaging parameters to obtain image data; a sensor digital control circuit for adjusting the imaging parameters according to control information and transmitting image data; a visible light-infrared image preprocessor for performing correction processing on the image data to obtain correction data; a visible light-infrared image heterogeneous processor for issuing control information and parsing the correction data to obtain parsing results; a data output interface circuit for receiving and outputting the parsing results; and a metal column array for three-dimensionally stacking and interconnecting the visible light-infrared detector array with the pixel-level readout circuit array, the visible light-infrared image preprocessor, and the visible light-infrared image heterogeneous processor.

[0006] According to an embodiment of the present invention, the programmable visible light-infrared vision chip also includes a data input interface circuit, a test module and a phase-locked loop; the data input interface circuit includes a test control signal interface circuit, which is used to obtain a test control signal and transmit the test control signal to the test module; the test module is used to perform functional testing according to the test control signal; the phase-locked loop is used to configure the main frequency clock of the chip.

[0007] According to an embodiment of the present invention, the data input interface circuit is also used to receive input parameters, which include chip parameters and initialization signals; wherein the input parameters are used to initialize the visible light-infrared detector array, the pixel-level readout circuit array, the sensor digital control circuit, the visible light-infrared image preprocessor and the visible light-infrared image heterogeneous processor respectively.

[0008] According to an embodiment of the present invention, the data input interface circuit also includes: a high-speed serializer / deserializer input interface circuit for receiving data input from outside the chip; a parallel input interface circuit and a serial input interface circuit for receiving chip parameters; and an initialization signal interface circuit for receiving an initialization signal.

[0009] According to an embodiment of the present invention, the visible light-infrared detector array includes M×N visible light-infrared detector units, which are used to receive light signals and convert the light signals into current signals, where M represents the number of rows and N represents the number of columns, and M and N are positive integers respectively.

[0010] According to an embodiment of the present invention, a pixel-level readout circuit array includes: M×N pixel-level readout circuit units, which are used to accumulate current signals and convert them into image data.

[0011] According to an embodiment of the present invention, the metal pillar array includes: M×N metal pillars, each metal pillar is connected to a visible light-infrared detector unit and a pixel-level readout circuit unit, and the metal pillar is used to transmit current signals.

[0012] According to an embodiment of the present invention, the sensor digital control circuit includes: a message parsing module, which is used to parse control information to obtain control instructions; a pulse generation module, which is used to send multiple pulse signals to the pixel-level readout circuit array according to the control instructions, and the pulse signals are used to adjust imaging parameters; and a result sending module, which is used to transmit image data to a visible light-infrared image preprocessor.

[0013] According to an embodiment of the present invention, the visible light-infrared image preprocessor includes: a multi-correction module, which is used to perform blind pixel correction and non-uniformity correction on image data to obtain second image data; a histogram stretching module, which is used to adaptively stretch the second image data to obtain third image data; and an image cropping module, which is used to crop the third image data to obtain corrected data.

[0014] According to an embodiment of the present invention, the histogram stretching module includes: a histogram statistical circuit, which is used to perform statistics on the grayscale of the second image data to obtain a statistical result; a threshold judgment circuit, which is used to determine a mapping threshold based on the statistical result; and a histogram mapping circuit, which is used to stretch the second image data based on the statistical result and the mapping threshold to obtain third image data.

[0015] According to an embodiment of the present invention, the image cropping module includes: a sequential storage circuit for sequentially storing the third image data; and a memory access cropping circuit for cropping the third image data to obtain corrected data using a custom memory access method.

[0016] According to an embodiment of the present invention, a visible light-infrared image heterogeneous processor includes: a microcontroller core for issuing control information and parsing small-scale serial correction data; a vector processor core for parsing large-scale parallel correction data; a neural network processor core for parsing neural network-related correction data; a data memory for storing correction data; and a processor data access circuit for realizing the input and output of correction data.

[0017] According to an embodiment of the present invention, the microcontroller core includes: a microcontroller instruction state machine for controlling the microcontroller core; a message encoding and decoding transceiver module for receiving messages from each module and sending control information; and a microcontroller processing circuit for parsing small-scale serial correction data.

[0018] According to an embodiment of the present invention, the message encoding and decoding transceiver module includes: a processor status monitoring module, which is used to monitor the internal environment of the chip to provide decision-making information to the microcontroller core; a peripheral message receiving module, which is used to receive messages from each module; and a peripheral message sending module, which is used to send control information.

[0019] According to an embodiment of the present invention, a vector processor core includes: a vector instruction state machine for controlling the vector processor core; a global processing circuit for parsing single-threaded correction data; and a vector processing circuit for parsing large-scale parallel correction data. According to an embodiment of the present invention, the global processing circuit includes: a global register for storing single-threaded correction data; and a global operation unit for performing operations related to the single-threaded correction data.

[0020] According to an embodiment of the present invention, the vector processing circuit includes: a vector register for storing large-scale correction data; and a vector operation unit for completing related operations on the large-scale correction data.

[0021] According to an embodiment of the present invention, the neural network processor core includes: a neural network instruction state machine for controlling the behavior of the neural network processor core; a neural network data access circuit for realizing the input and output of correction data; a weight storage circuit for storing and receiving weight data required for neural network calculations; a feature map storage circuit for storing and receiving feature map data required for neural network calculations; a neural network processing array for performing convolution calculations on correction data related to the neural network; an accumulator for performing accumulation operations on correction data related to the neural network; and a neural network post-processing circuit for post-processing correction data related to the neural network.

[0022] According to an embodiment of the present invention, the neural network data access circuit includes: a weight storage input transporter, used to transfer weight data from the data storage device to the weight storage circuit; a feature map storage input transporter, used to transfer feature map data from the data storage device to the feature map storage circuit; and a neural network post-processing output transporter, used to transfer the results of the neural network post-processing circuit operation back to the data storage device.

[0023] According to an embodiment of the present invention, the weight storage circuit includes: a weight memory for storing weight data required for neural network calculation; and a weight buffer output module for caching and outputting data in the weight memory to a neural network processing array.

[0024] According to an embodiment of the present invention, the feature map storage circuit includes: a feature map memory for storing feature map data required for neural network calculation; and a feature map buffer output module for caching the data in the feature map memory and outputting it to the neural network processing array.

[0025] According to an embodiment of the present invention, a neural network processing array includes: Q×Q neural network processing units, which are used to complete a single operation of a neural network convolution operation, where Q represents the number of rows and columns and Q is a positive integer; a programmable connector, which is used to dynamically combine the neural network processing units to complete the convolution operation of the neural network.

[0026] According to an embodiment of the present invention, the neural network processing unit includes: a weight register for storing weight data to be calculated; a feature map register for storing feature map data to be calculated; a calculation unit for performing calculations based on the weight data and feature map data; and a result register for storing result data after the calculation unit completes the calculation.

[0027] According to an embodiment of the present invention, the data output interface circuit 20 includes: a high-speed serializer / deserializer output interface circuit and a parallel output interface circuit for receiving and outputting parsing results; and a message output interface circuit for sending custom messages.

[0028] The method of the present invention realizes large-scale visible light-infrared visual imaging tasks through three-dimensional heterogeneous integration of a visible light-infrared detector array and a pixel-level readout circuit array. It also compensates for the blind pixels and non-uniformity defects inherent in the visible light-infrared detector array through a visible light-infrared image preprocessor. It also realizes the analysis of different processing tasks through a visible light-infrared image heterogeneous processor. It has high flexibility and solves the problems of blind pixels and non-uniformity existing in current infrared vision chips, as well as the lack of flexibility in analyzing different processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0030] Figure 1 Schematically shows one of the structural diagrams of a programmable visible light infrared vision chip according to an embodiment of the present invention;

[0031] Figure 2 The second structural diagram of the programmable visible light infrared vision chip according to an embodiment of the present invention is schematically shown;

[0032] Figure 3 Schematically shows a structural diagram of the sensing and readout part according to an embodiment of the present invention;

[0033] Figure 4 A schematic diagram illustrating the structure of a visible light-infrared image preprocessor according to an embodiment of the present invention is shown;

[0034] Figure 5 A schematic diagram illustrating the structure of a visible light-infrared image heterogeneous processor according to an embodiment of the present invention is shown;

[0035] Figure 6 The data flow diagram of the programmable visible light infrared vision chip according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0037] It should be noted that in the drawings or descriptions of the specification, similar or identical parts use the same figure numbers. The technical features of the various embodiments exemplified in the specification can be freely combined to form new solutions without conflict, and in the drawings, the shape or thickness of the embodiments can be expanded and simplified or conveniently indicated. Furthermore, the elements or implementations not shown or described in the drawings are forms known to those of ordinary skill in the art. In addition, although this document may provide examples of parameters containing specific values, it should be understood that the parameters do not need to be exactly equal to the corresponding values, but can be approximated to the corresponding values ​​within an acceptable error tolerance or design constraint.

[0038] Unless there are technical obstacles or contradictions, the above-mentioned various embodiments of the present invention can be freely combined to form other embodiments, and these other embodiments are all within the protection scope of the present invention.

[0039] Although the present invention is described in conjunction with the accompanying drawings, the embodiments disclosed in the drawings are intended to illustrate preferred embodiments of the present invention and are not to be construed as limiting the present invention. The dimensions and proportions in the drawings are merely illustrative and are not to be construed as limiting the present invention.

[0040] Although certain embodiments of the present general inventive concept have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the present general inventive concept.

[0041] Figure 1 One of the structural diagrams of a programmable visible light infrared vision chip according to an embodiment of the present invention is schematically shown.

[0042] like Figure 1 As shown, an embodiment of the present invention provides a programmable visible light infrared vision chip, including: a visible light-infrared detector array 30, used to detect light signals in the visible light infrared band and convert the light signals into current signals; a pixel-level readout circuit array 40, used to perform image conversion on the current signal according to imaging parameters to obtain image data; a sensor digital control circuit 60, used to adjust the imaging parameters according to control information and transmit image data; a visible light-infrared image preprocessor 70, used to correct the image data to obtain corrected data; a visible light-infrared image heterogeneous processor 80, used to send control information and parse the corrected data to obtain parsing results; a data output interface circuit 20, used to receive and output the parsing results; a metal column array 50, used to three-dimensionally stack and interconnect the visible light-infrared detector array 30 with the pixel-level readout circuit array 40, the visible light-infrared image preprocessor 70 and the visible light-infrared image heterogeneous processor 80.

[0043] The method provided by the embodiments of the present invention achieves large-scale visible-infrared visual imaging tasks through the three-dimensional heterogeneous integration of a visible-infrared detector array and a pixel-level readout circuit array. The visible-infrared image preprocessor compensates for the inherent blind pixels and non-uniformity defects of the visible-infrared detector array. The heterogeneous visible-infrared image processor enables the parsing of different processing tasks with high flexibility. This method can thus address the blind pixels and non-uniformity issues inherent in current infrared vision chips, as well as the inability to parse different processing tasks with limited flexibility.

[0044] Figure 2 The second structural diagram of the programmable visible light infrared vision chip according to an embodiment of the present invention is schematically shown.

[0045] like Figure 2 As shown, the programmable visible light-infrared vision chip of this embodiment also includes a data input interface circuit 10, a test module 100 and a phase-locked loop (101); the data input interface circuit 10 includes a test control signal interface circuit 140 for obtaining a test control signal and transmitting the test control signal to the test module 100; the test module 100 is used to perform functional testing according to the test control signal; and the phase-locked loop 101 is used to configure the main frequency clock of the chip operation.

[0046] In some embodiments, when the chip is in the test mode, the test module 100 sends information about key chip registers to the outside of the chip for rapid detection of chip validity and location of failed modules.

[0047] In some embodiments, the data input interface circuit 10 is also used to receive input parameters, which include chip parameters and initialization signals; wherein the input parameters are used to initialize the visible light-infrared detector array 30, the pixel-level readout circuit array 40, the sensor digital control circuit 60, the visible light-infrared image preprocessor 70 and the visible light-infrared image heterogeneous processor 80 respectively.

[0048] Please continue reading Figure 2 The data input interface circuit 10 also includes: a high-speed serializer / deserializer input interface circuit 110 for receiving data input from outside the chip; a parallel input interface circuit 120 and a serial input interface circuit 130 for receiving chip parameters; and an initialization signal interface circuit 150 for receiving an initialization signal.

[0049] Figure 3 The structure of the sensing and readout part according to an embodiment of the present invention is schematically shown.

[0050] like Figure 3As shown, the visible light-infrared detector array 30 of this embodiment includes M×N visible light-infrared detector units 310, which are used to receive light signals and convert the light signals into current signals, where M represents the number of rows and N represents the number of columns, and M and N are positive integers respectively.

[0051] Please continue reading Figure 3 The pixel-level readout circuit array 40 includes: M×N pixel-level readout circuit units 410, which are used to accumulate current signals and convert them into image data.

[0052] The pixel-level readout circuit unit 410 may accumulate the current signal generated by the detector unit and convert it into a pixel value in a P-bit digital circuit signal format for subsequent digital circuit processing, where P is a positive integer.

[0053] The metal pillar array 50 includes M×N metal pillars, each of which is connected to a visible light-infrared detector unit 310 and a pixel-level readout circuit unit 410 , and is used to transmit current signals.

[0054] In some embodiments, the programmable visible light infrared vision chip may include two sub-chips, the upper layer is a sub-chip composed of a visible light-infrared detector array 30, and the lower layer is a second sub-chip composed of a pixel-level readout circuit array 40 and other circuits.

[0055] Please continue reading Figure 3 The sensor digital control circuit 60 includes: a message parsing module 610, which is used to parse control information to obtain control instructions; a pulse generating module 620, which is used to send multiple pulse signals to the pixel-level readout circuit array 40 according to the control instructions, and the pulse signals are used to adjust imaging parameters; and a result sending module 630, which is used to transmit image data to the visible light-infrared image preprocessor 70.

[0056] Figure 4 The structure of a visible light-infrared image preprocessor according to an embodiment of the present invention is schematically shown.

[0057] like Figure 4 As shown, the visible light-infrared image preprocessor 70 includes: a multi-correction module 710, which is used to perform blind pixel correction and non-uniformity correction on the image data to obtain second image data; a histogram stretching module 720, which is used to adaptively stretch the second image data to obtain third image data; and an image cropping module 730, which is used to crop the third image data to obtain corrected data.

[0058] Among them, the histogram stretching module 720 includes: a histogram statistics circuit 7210, which is used to perform statistics on the grayscale of the second image data to obtain statistical results; a threshold judgment circuit 7220, which is used to determine the mapping threshold according to the statistical results; and a histogram mapping circuit 7230, which is used to stretch the second image data according to the statistical results and the mapping threshold to obtain third image data.

[0059] In some embodiments, the image cropping module 730 includes: a sequential storage circuit 7310 for sequentially storing the third image data; and a memory access cropping circuit 7320 for cropping the third image data to obtain corrected data through a custom memory access method.

[0060] Figure 5 The figure schematically shows a structural diagram of a visible light-infrared image heterogeneous processor according to an embodiment of the present invention.

[0061] like Figure 5 As shown, the visible light-infrared image heterogeneous processor 80 includes: a microcontroller core 810, which is used to issue control information and parse small-scale serial correction data; a vector processor core 820, which is used to parse large-scale parallel correction data; a neural network processor core 830, which is used to parse neural network-related correction data; a data memory 840, which is used to store correction data; and a processor data access circuit 850, which is used to realize the input and output of correction data.

[0062] The microcontroller core 810 includes: a microcontroller instruction state machine 8110 for controlling the microcontroller core 810; a message encoding and decoding transceiver module 8120 for receiving messages from various modules and sending control information; and a microcontroller processing circuit 8130 for parsing small-scale serial correction data.

[0063] In some embodiments, the message encoding and decoding transceiver module 8120 includes: a processor status monitoring module 81201, which is used to monitor the internal environment of the chip to provide decision information to the microcontroller core 810; a peripheral message receiving module 81202, which is used to receive messages from each module; and a peripheral message sending module 81203, which is used to send control information.

[0064] In some embodiments, the vector processor core 820 includes: a vector instruction state machine 8210 for controlling the vector processor core 820; a global processing circuit 8220 for parsing single-threaded correction data; and a vector processing circuit 8230 for parsing large-scale parallel correction data.

[0065] In some embodiments, the global processing circuit 8220 includes: a global register 82201 for storing correction data of a single thread; and a global operation unit 82202 for completing related operations of the correction data of a single thread.

[0066] In some embodiments, the vector processing circuit 8230 includes: a vector register 82301 for storing large-scale correction data; and a vector operation unit 82302 for performing related operations on the large-scale correction data.

[0067] In some embodiments, the neural network processor core 830 includes: a neural network instruction state machine 8310, which is used to control the behavior of the neural network processor core 830; a neural network data access circuit 8320, which is used to realize the input and output of correction data; a weight storage circuit 8330, which is used to store and receive weight data required for neural network calculations; a feature map storage circuit 8340, which is used to store and receive feature map data required for neural network calculations; a neural network processing array 8350, which is used to perform convolution calculations on correction data related to the neural network; an accumulator 8360, which is used to perform accumulation operations on correction data related to the neural network; and a neural network post-processing circuit 8370, which is used to post-process the correction data related to the neural network.

[0068] In some embodiments, the neural network data access circuit 8320 includes: a weight storage input transport, used to transfer weight data from the data storage 840 to the weight storage circuit 8330; a feature map storage input transport, used to transfer feature map data from the data storage 840 to the feature map storage circuit 8340; and a neural network post-processing output transport, used to transfer the results of the operation completed by the neural network post-processing circuit 8370 back to the data storage 840.

[0069] In some embodiments, the weight storage circuit 8330 includes: a weight memory for storing weight data required for neural network calculations; and a weight buffer output module for caching and outputting data in the weight memory to a neural network processing array.

[0070] In some embodiments, the feature map storage circuit 8340 includes: a feature map memory for storing feature map data required for neural network calculations; and a feature map buffer output module for caching and outputting data in the feature map memory to the neural network processing array 8350.

[0071] In some embodiments, the neural network processing array 8350 includes: Q×Q neural network processing units 83501, which are used to complete a single operation of a neural network convolution operation, where Q represents the number of rows and columns, and Q is a positive integer; a programmable connector 83502, which is used to dynamically combine the neural network processing units to complete the convolution operation of the neural network.

[0072] In some embodiments, the neural network processing unit includes: a weight register for storing weight data to be calculated; a feature map register for storing feature map data to be calculated; a calculation unit for performing calculations based on the weight data and feature map data; and a result register for storing result data after the calculation unit completes the calculation.

[0073] In some embodiments, the data output interface circuit 20 includes: a high-speed serializer / deserializer output interface circuit 210 and a parallel output interface circuit 220 for receiving and outputting parsing results; and a message output interface circuit 230 for sending custom messages.

[0074] Figure 6 The data flow diagram of the programmable visible light infrared vision chip according to an embodiment of the present invention is schematically shown.

[0075] like Figure 6 As shown, the data flow of the embodiment of the present invention includes steps S610 to S650.

[0076] Step S610: performing a global reset on the programmable visible light infrared vision chip.

[0077] Step S620: Loading infrared heterogeneous image processor instructions from the initialization signal interface.

[0078] Step S640: Load algorithm parameters from the serial input interface.

[0079] In step S650 , the microcontroller performs a parsing operation.

[0080] Step S660: the programmable visible light infrared vision chip ends its operation.

[0081] Furthermore, the above step S640 may further include:

[0082] Sub-step S6401: interact with the outside through the input interface.

[0083] Sub-step S6402: performing a parsing operation by the vector processor core.

[0084] Sub-step S6403, performing parsing operations through the neural network processor core.

[0085] After a global reset, the programmable visible-light infrared vision chip of this embodiment of the present invention loads visible-light and infrared heterogeneous image processor instructions from the initialization signal interface circuit 150, then loads algorithm parameters from the serial input interface circuit 130. After completion, the microcontroller begins executing instructions. During microcontroller operation, the rest of the chip may be called upon to perform the following three operations: interacting with the external world through input / output interface circuits, executing instructions from the vector processor core, and executing instructions from the neural network processor core. After the microcontroller completes its operation, the chip terminates.

[0086] It is understood that the specific order or hierarchy of steps in the processes disclosed is an example of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged without departing from the scope of the present invention.

[0087] It should also be noted that directional terms such as "upper," "lower," "front," "back," "left," and "right" mentioned in the embodiments are merely references to the directions in the accompanying drawings and are not intended to limit the scope of protection of the present invention. Throughout the drawings, identical elements are represented by identical or similar reference numerals. Conventional structures or configurations will be omitted if they may cause confusion in understanding the present invention. Furthermore, the shapes, sizes, and positional relationships of the components in the drawings do not reflect their actual sizes, proportions, or actual positional relationships.

[0088] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. With respect to the term "comprising" used in the specification, the word is encompassed in a manner similar to the term "including", as explained when "including" is used as a transitional word.

[0089] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A programmable visible light infrared vision chip, characterized in that: include: A visible light-infrared detector array (30) is used to detect optical signals in the visible light-infrared band and convert the optical signals into current signals; A pixel-level readout circuit array (40) is used to perform image conversion on the current signal according to imaging parameters to obtain image data; A sensor digital control circuit (60) is used to adjust the imaging parameters according to the control information and transmit the image data; A visible light-infrared image preprocessor (70) is used to perform correction processing on the image data to obtain corrected data; A visible light-infrared image heterogeneous processor (80) is used to issue control information and parse the correction data to obtain a parsing result; A data output interface circuit (20) is used to receive and output the analysis result; A metal pillar array (50) is used to three-dimensionally stack and interconnect the visible light-infrared detector array (30), the pixel-level readout circuit array (40), the visible light-infrared image preprocessor (70), and the visible light-infrared image heterogeneous processor (80); Wherein, the visible light-infrared image preprocessor (70) includes: A multiple correction module (710) is used to perform blind pixel correction and non-uniformity correction processing on the image data to obtain second image data; a histogram stretching module (720), configured to adaptively stretch the second image data to obtain third image data; An image cropping module (730), configured to crop the third image data to obtain the corrected data; The visible light-infrared image heterogeneous processor (80) includes: A microcontroller core (810) is used to issue the control information and parse the small-scale serial correction data; A vector processor core (820) for parsing the correction data in large-scale parallel; A neural network processor core (830) for parsing the correction data related to the neural network; A data storage device (840) for storing the correction data; The processor data access circuit (850) is used to implement the input and output of the correction data.

2. The programmable visible light infrared vision chip according to claim 1, characterized in that: It also includes a data input interface circuit (10), a test module (100) and a phase-locked loop (101); The data input interface circuit (10) includes a test control signal interface circuit (140) for acquiring a test control signal and transmitting the test control signal to the test module (100); The test module (100) is used to perform a functional test according to the test control signal; The phase-locked loop (101) is used to configure the main frequency clock of the chip operation.

3. The programmable visible light infrared vision chip according to claim 2, characterized in that: The data input interface circuit (10) is further used to receive input parameters, the input parameters including chip parameters and initialization signals; The input parameters are used to respectively initialize the visible light-infrared detector array (30), the pixel-level readout circuit array (40), the sensor digital control circuit (60), the visible light-infrared image preprocessor (70), and the visible light-infrared image heterogeneous processor (80).

4. The programmable visible light infrared vision chip according to claim 3, characterized in that: The data input interface circuit (10) further includes: A high-speed serializer / deserializer input interface circuit (110) for receiving data input from outside the chip; A parallel input interface circuit (120) and a serial input interface circuit (130) for receiving the chip parameters; and an initialization signal interface circuit (150) for receiving the initialization signal.

5. The programmable visible light infrared vision chip according to claim 1, characterized in that: The visible light-infrared detector array (30) comprises: M×N visible light-infrared detector units (310) are used to receive the optical signal and convert the optical signal into the current signal, wherein M represents the number of rows, N represents the number of columns, and M and N are positive integers respectively.

6. The programmable visible light infrared vision chip according to claim 5, characterized in that: The pixel-level readout circuit array (40) comprises: M×N pixel-level readout circuit units (410) are used to accumulate the current signal and convert it into the image data.

7. The programmable visible light infrared vision chip according to claim 6, characterized in that: The metal pillar array (50) comprises: M×N metal pillars, each of the metal pillars is connected to a visible light-infrared detector unit (310) and a pixel-level readout circuit unit (410), and the metal pillars are used to transmit the current signal.

8. The programmable visible light infrared vision chip according to claim 1, characterized in that: The sensor digital control circuit (60) comprises: A message parsing module (610) is used to parse the control information to obtain a control instruction; A pulse generating module (620) is used to send a plurality of pulse signals to the pixel-level readout circuit array (40) according to the control instruction, wherein the pulse signals are used to adjust the imaging parameters; A result sending module (630) is used to transmit the image data to the visible light-infrared image preprocessor (70).

9. The programmable visible light infrared vision chip according to claim 1, characterized in that: The histogram stretching module (720) includes: a histogram statistics circuit (7210), configured to perform statistics on the grayscale of the second image data to obtain a statistical result; A threshold determination circuit (7220), configured to determine a mapping threshold based on the statistical result; A histogram mapping circuit (7230) is configured to stretch the second image data according to the statistical result and the mapping threshold to obtain third image data.

10. The programmable visible light infrared vision chip according to claim 1, characterized in that: The image cropping module (730) includes: A sequential storage circuit (7310), configured to store the third image data in sequence; The memory access type cropping circuit (7320) is used to crop the third image data through a customized memory access method to obtain the corrected data.

11. The programmable visible light infrared vision chip according to claim 1, characterized in that: The microcontroller core (810) includes: A microcontroller instruction state machine (8110) for controlling the microcontroller core (810); Message encoding and decoding transceiver module (8120), used for receiving messages from various modules and sending the control information; The microcontroller processing circuit (8130) is used for parsing the small-scale serial correction data.

12. The programmable visible light infrared vision chip according to claim 11, characterized in that: The message encoding and decoding transceiver module (8120) includes: A processor status monitoring module (81201) is used to monitor the internal environment of the chip to provide decision-making information to the microcontroller core (810); Peripheral message receiving module (81202), used to receive messages from various modules; The peripheral message sending module (81203) is used to send the control information.

13. The programmable visible light infrared vision chip according to claim 1, characterized in that: The vector processor core (820) includes: A vector instruction state machine (8210) for controlling the vector processor core (820); A global processing circuit (8220) is used to parse the correction data of a single thread; The vector processing circuit (8230) is used for parsing the correction data in large-scale parallel.

14. The programmable visible light infrared vision chip according to claim 13, characterized in that: The global processing circuit (8220) comprises: A global register (82201), used to store the correction data of a single thread; The global operation unit (82202) is used to complete the relevant operations of the correction data of a single thread.

15. The programmable visible light infrared vision chip according to claim 13, characterized in that: The vector processing circuit (8230) comprises: Vector register (82301), used to store large-scale correction data; The vector operation unit (82302) is used to complete large-scale related operations of the correction data.

16. The programmable visible light infrared vision chip according to claim 1, characterized in that: The neural network processor core (830) includes: A neural network instruction state machine (8310) for controlling the behavior of the neural network processor core (830); A neural network data access circuit (8320) is used to implement input and output of the correction data; A weight storage circuit (8330) for storing and sending and receiving weight data required for neural network calculations; A feature graph storage circuit (8340) for storing and receiving feature graph data required for neural network calculations; A neural network processing array (8350) for performing convolution calculations on the correction data associated with the neural network; An accumulator (8360) is used to perform accumulation operation on the correction data related to the neural network; and a neural network post-processing circuit (8370) for post-processing the correction data related to the neural network.

17. The programmable visible light infrared vision chip according to claim 16, characterized in that: The neural network data access circuit (8320) includes: A weight storage input transporter, used to transfer weight data from the data storage (840) to the weight storage circuit (8330); A feature map storage input transporter, used to transfer feature map data from the data storage (840) to the feature map storage circuit (8340); The neural network post-processing output transporter is used to transmit the results of the operation completed by the neural network post-processing circuit (8370) back to the data storage (840).

18. The programmable visible light infrared vision chip according to claim 16, characterized in that: The weight storage circuit (8330) includes: Weight memory, used to store weight data required for neural network calculations; The weight buffer output module is used to cache the data in the weight memory and output it to the neural network processing array.

19. The programmable visible light infrared vision chip according to claim 16, characterized in that: The characteristic graph storage circuit (8340) comprises: Feature map memory, used to store feature map data required for neural network calculation; A feature map buffer output module is used to cache the data in the feature map memory and output it to the neural network processing array (8350).

20. The programmable visible light infrared vision chip according to claim 16, characterized in that: The neural network processing array (8350) includes: Q×Q neural network processing units (83501), used to complete a single operation of a neural network convolution operation, where Q represents the number of rows and columns and Q is a positive integer; The programmable connector (83502) is used to dynamically combine neural network processing units to complete the convolution operation of the neural network.

21. The programmable visible light infrared vision chip according to claim 20, characterized in that: The neural network processing unit includes: Weight register, used to store weight data to be calculated; Feature map register, used to store feature map data to be calculated; a calculation unit, configured to perform calculations based on the weight data and the feature map data; The result register is used to store the result data after the calculation unit completes the operation.

22. The programmable visible light infrared vision chip according to claim 1, characterized in that: The data output interface circuit (20) comprises: A high-speed serializer / deserializer output interface circuit (210) and a parallel output interface circuit (220) are used to receive and output the parsing result; The message output interface circuit (230) is used to send a custom message.

Images