Reconfigurable convolution weight loading and reading system of sensing and computing integrated visual chip

Abstract

The invention relates to the technical field of photoelectric sensing and integrated circuits, and particularly provides a reconfigurable convolution weight loading and reading system of a sensing and computing integrated visual chip, which comprises a reconfigurable convolution weight loading system, a sensing and computing integrated pixel array, a column merging reading system and a time sequence control module, the reconfigurable convolution weight loading system is used for configuring a multi-scale convolution kernel, converting a digital weight into an analog voltage signal, and loading the analog voltage signal to the sensing and computing integrated pixel array by adopting a reconfigurable convolution weight loading mechanism based on a 13 * 13 weight loading array; the sensing and computing integrated pixel array is used for carrying out pixel-level visual information sensing and parallel convolution calculation on the analog voltage signals, the column merging reading system is used for reading a parallel convolution calculation result of the sensing and computing integrated pixel array, and the time sequence control module is used for providing time sequence control signals. According to the method and the device, multi-scale convolution kernel configuration is realized, convolution results of different convolution kernel sizes and step lengths are read, and diversified visual processing requirements are met.

Description

Technical Field

[0001] This application relates to the field of optoelectronic sensing and integrated circuit technology, specifically to a reconfigurable convolution weight loading and readout system for a sensing and computing integrated vision chip. Background Technology

[0002] Image sensors, as the core sensing devices in camera modules, can efficiently convert light signals into electrical signals, while convolutional neural networks significantly improve the processing efficiency of machine vision tasks through features such as local connectivity and weight sharing. However, traditional vision processing systems based on the von Neumann architecture generally face severe data transfer bottlenecks. This not only leads to high processing latency but also imposes a significant energy consumption burden.

[0003] To overcome this technological bottleneck, the integrated vision chip has emerged. This chip achieves synchronous processing of photoelectric conversion and feature extraction by directly embedding convolution calculation function within the pixel array, reducing energy consumption caused by redundant data transmission and also reducing processing latency. Therefore, it has important application value in fields with strong real-time and low power consumption, such as military navigation, robotics, autonomous driving, and smart homes.

[0004] Currently, sensor-computing integration technology mainly presents three technical paradigms: Near-Sensor Computing (NSC) (see Reference 1: T.-H. Hsu et al., "A 0.5-V Real-Time Computational CMOS Image Sensor With Programmable Kernel for Feature Extraction", published in IEEE Journal of Solid-State Circuits, Vol. 56, No. 5, pp. 1588-1596, 2021), In-Sensor Computing (ISC) (see Reference 2: H. Xu et al., "MACSen: A Processing-In-Sensor Architecture Integrating MAC Operations Into Image Sensor for Ultra-Low-Power BNN-Based Intelligent Visual Perception", published in IEEE Transactions on Circuits and Systems II: Express Briefs, Vol. 68, No. 2, pp. 627-631, 2021), and In-Pixel Computing (ISC). (Reference 3: Datta, G., Kundu, S., Yin, Z., et al., “A processing-in-pixel-in-memory paradigm for resource-constrained TinyML applications”, Scientific Reports (Sci Rep), Vol. 12, No. 14396, 2022).

[0005] Among them, the in-sensor computing technology based on intra-pixel computation achieves pixel-level parallel processing by equipping each pixel with an independent analog domain computing circuit, demonstrating advantages in computational flexibility. This approach can perform fine-grained preliminary processing of visual information, such as edge detection and quality filtering, while significantly reducing data transmission requirements. It improves processing speed, reduces visual system latency, and optimizes energy efficiency. However, existing fixed-architecture integrated sensing and computing circuits, due to their rigid computational modes, struggle to adapt to the dynamic requirements of different neural networks regarding convolution kernel size and stride, which to some extent limits their application potential in complex visual tasks.

[0006] For energy- and latency-sensitive applications such as edge computing, reconfigurable computing architectures have become a key solution to unleash the advantages of integrated sensing and computing technologies. However, existing research on reconfigurable architectures for integrated sensing and computing vision chips still faces several key technical bottlenecks. At the hardware design level, in order to achieve variable convolutional kernels with a large range of variations (such as 2×2 to 64×64 sizes), (Reference 4: M. Lefebvre, L. Moreau, R. Dekimpe and D. Bol, "7.7A 0.2-to-3.6TOPS / W Programmable Convolutional Imager SoC with In-Sensor Current-Domain Ternary-Weighted MAC Operations for Feature Extraction and Region-of-Interest Detection", published in 2021 IEEE International Solid-State Circuits) The conference (ISSCC), pp. 118-120, 2021, argues that the excessive pursuit of flexibility in convolutional kernel size leads to a significant increase in circuit area and power consumption, while practical deep learning applications only need to support a limited number of standard convolutional kernel sizes, such as 3×3, 5×5, and 7×7. In terms of computational efficiency, the serial weight loading mechanism limits the improvement of processing speed and makes it difficult to meet the needs of high frame rate scenarios. In terms of functional scalability, the static reconstruction mechanism cannot support dynamic model switching in multi-task scenarios.

[0007] More importantly, the weight loading and readout circuits of current intra-pixel computational (pixel-level) integrated vision chips are still mainly designed for fixed convolution kernel sizes, lacking specific support for reconstructable convolutions. This design limitation not only restricts the deep integration of the intra-pixel computation paradigm with reconstructable convolutions, but also limits the performance of integrated vision chips in vision tasks.

[0008] Furthermore, research on reconfigurable convolution weight loading circuits for integrated sensor-memory-computing pixel arrays using novel non-silicon materials such as InGaAs and perovskite is almost nonexistent. On the other hand, while existing readout circuits can achieve basic focal plane readout functions, they have not optimized the circuit structure for the parallel processing characteristics of intra-pixel computation and the requirements of reconfigurable convolution. This not only fails to fully leverage the advantages of integrated sensor-memory-computing technology in reducing data bandwidth, latency, and power consumption, but also leads to unnecessary circuit area overhead and system latency.

[0009] Therefore, it is necessary to design a reconfigurable convolution weight loading and readout circuit for pixel-level integrated vision chips to solve these key problems, and it is expected to achieve a comprehensive improvement in the flexibility, latency and energy efficiency of vision processing systems, laying a technical foundation for the development of next-generation intelligent vision chips. Summary of the Invention

[0010] To address the shortcomings of existing technologies, the purpose of this application is to provide a reconfigurable convolutional weight loading and readout circuit system for an integrated vision chip.

[0011] A first aspect of this application provides a reconfigurable convolutional weight loading and readout circuit system for an integrated sensing and computing vision chip, comprising:

[0012] The system comprises a reconfigurable convolutional weight loading system, a sensor-computer integrated pixel array, a column merging readout system, and a timing control module. The reconfigurable convolutional weight loading system is connected to the sensor-computer integrated pixel array, which is connected to the column merging readout system. The timing control module is connected to both the reconfigurable convolutional weight loading system and the column merging readout system. The reconfigurable convolutional weight loading system configures multi-scale convolutional kernels and converts digital weights into analog voltage signals. It uses a reconfigurable convolutional weight loading mechanism based on a 13×13 weight loading array to load the analog voltage signals onto the sensor-computer integrated pixel array. The sensor-computer integrated pixel array performs pixel-level visual information perception and parallel convolution calculation on the analog voltage signals. The column merging readout system reads out the parallel convolution calculation results of the sensor-computer integrated pixel array. The timing control module provides timing control signals.

[0013] Optionally, the reconfigurable convolution weight loading system includes a weight voltage mapping module, a weight loading module, and a row selection module. The weight voltage mapping module is used to convert the digital weights into the analog voltage signal, the weight loading module is used to dynamically load multi-scale convolution kernels, and the row selection module is used to load the analog voltage signal.

[0014] Optionally, the weighted voltage mapping module includes a digital weight storage unit, a digital-to-analog converter circuit, and a voltage regulation and drive circuit;

[0015] The digital weight storage unit includes a memory and a register. The memory is used to store quantized digital weights, and the register is used to store digital weights to be processed.

[0016] The digital-to-analog converter circuit is used to convert the digital weights into the analog voltage signal;

[0017] The voltage regulation and drive circuit includes a noise suppression circuit, a programmable voltage bias circuit, and an operational amplifier. The noise suppression circuit is used for correlated double sampling, the programmable voltage bias circuit is used for dynamically adjusting the reference voltage or bias point, and the operational amplifier is used to amplify the input signal to a preset level.

[0018] Optionally, the weight loading module is used to arrange multi-scale convolutional kernels on a preset basic weight loading array using a preset weight arrangement method, and to construct an integrated sensing and computing pixel array using the preset basic weight loading array as the basic building unit and a tile-type expansion strategy. The row weights and column weights of the preset basic weight loading array have some overlapping areas. The preset basic weight loading array uses a reconfigurable convolutional weight loading mechanism based on a 13×13 weight loading array to load the analog voltage signal onto the integrated sensing and computing pixel array.

[0019] Optionally, the row selection module includes M row selection switches, each row selection switch corresponding to control each row of pixels in the integrated sensing and computing pixel array.

[0020] Optionally, the column merging and readout system includes a column merging module, a sample / hold module, and a multiplexing module. The column merging module is used to merge current signals according to the reconfigurable convolution kernel size and read out the analog voltage signal. The sample / hold module is used to transmit the analog voltage signal to the multiplexing module, and the multiplexing module is used to switch signal paths.

[0021] Optionally, the column merging module includes N-1 column merging switches and N sets of integrating capacitors. The input terminal of each set of integrating capacitors is connected to the column line of the integrated pixel array, and each switch is connected to the adjacent column line of the integrated pixel array.

[0022] Optionally, the sample / hold module includes N sets of sample / hold circuits, each set of sample / hold circuits being coupled to each set of integrating capacitors.

[0023] Optionally, the multiplexing module includes a multiplexer, and the input of each multiplexer is connected to a preset number of sample / hold circuits.

[0024] Optionally, the integrated pixel array for sensing and computing adopts a CMOS process integration scheme or a heterogeneous integration scheme.

[0025] The reconfigurable convolution weight loading and readout circuit system of the integrated sensing and computing vision chip of this application adopts a reconfigurable convolution weight loading system to configure multi-scale convolution kernels and convert digital weights into analog voltage signals. The integrated sensing and computing pixel array realizes convolution calculations with convolution kernels of different scales and strides. The column merging readout system flexibly reads out the convolution calculation results with convolution kernels of different scales and strides, meeting the dynamic requirements of different neural networks for convolution kernel size and convolution stride, realizing diversified vision processing and improving hardware resource utilization.

[0026] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0027] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0028] Figure 1 This is a schematic diagram illustrating the structure of a reconfigurable convolutional weight loading and readout system for a sensor-memory integrated vision chip according to an exemplary embodiment.

[0029] Figure 2 This is a schematic diagram of the circuit structure of a weighted voltage mapping module according to an exemplary embodiment.

[0030] Figure 3 This is a schematic diagram illustrating the operation of a weight loading module according to an exemplary embodiment.

[0031] Figure 4 This is a schematic diagram illustrating the connection rule of a 3×3 convolution kernel in a 13×13 weighted array according to an exemplary embodiment.

[0032] Figure 5 This is a schematic diagram illustrating the connection rules in a 13×13 weighted array after a 3×3 convolution kernel slides to the right once, according to an exemplary embodiment.

[0033] Figure 6 This is a schematic diagram illustrating the connection rule of a 5×5 convolution kernel in a 13×13 weighted array according to an exemplary embodiment.

[0034] Figure 7 This is a schematic diagram illustrating the connection rules of a 7×7 convolution kernel in a 13×13 weighted array according to an exemplary embodiment.

[0035] Figure 8 This is a schematic diagram of the circuit structure of a row selection module according to an exemplary embodiment.

[0036] Figure 9This is a schematic diagram of the circuit structure of a row selection switch according to an exemplary embodiment.

[0037] Figure 10 This is a schematic diagram illustrating the collaborative operation of a sensor-computer integrated pixel array and column merging readout system according to an exemplary embodiment.

[0038] Figure 11 This is a schematic diagram illustrating the collaborative operation of a sample / hold module and a multiplexing module according to an exemplary embodiment.

[0039] In the picture:

[0040] A reconfigurable convolutional weight loading and readout system for a 100-bit integrated vision chip;

[0041] 110 is a reconfigurable convolutional weight loading system;

[0042] 111 is the weighted voltage mapping module; 111-1 is the digital weight storage unit; 111-2 is the digital-to-analog conversion circuit; 111-3 is the voltage regulation and drive circuit;

[0043] 112 is the weight loading module; 112-1 is a 13×13 weight loading array;

[0044] 113 is the row selection module; 113-1 is the row selection switch;

[0045] 120 represents a pixel array integrating sensing and computing; 120-1 represents a pixel.

[0046] 130-column merge readout system;

[0047] 131 is the column merging module; 131-1 is the column merging switch; 131-2 is the integrating capacitor;

[0048] 132 is the sample / hold module; 132-1 is the sample / hold circuit;

[0049] 133 is a multiplexing module; 133-1 is a multiplexer;

[0050] 140 is the timing control module.

[0051] Figure 4 In the middle (a), it represents a 3×3 convolution kernel. Figure 4 In the middle (b), the spatial distribution of 6×6 3×3 convolutional kernels in the 13×13 weighted array 112-1 is shown, and the gray filled area represents the weight overlap area between adjacent rows / columns of convolutional kernels.

[0052] Figure 5 In the middle (a), the kernel is represented after the 3×3 kernel has been slid to the right once. Figure 5In the middle (b), the spatial distribution of 6×6 3×3 convolution kernels after sliding to the right once is shown in the 13×13 weight loading array 112-1. The gray filled area represents the weight overlap area between adjacent rows / columns of convolution kernels.

[0053] Figure 6 In the middle (a), it represents a 5×5 convolution kernel. Figure 6 In the middle (b), the spatial distribution of 3×3 5×5 convolutional kernels in the 13×13 weighted array 112-1 is shown, and the gray filled area represents the weight overlap area between adjacent rows / columns of convolutional kernels.

[0054] Figure 7 In the middle (a), it represents a 7×7 convolution kernel. Figure 7 In the middle (b), the spatial distribution of 2×2 7×7 convolutional kernels in the 13×13 weighted array 112-1 is shown, and the gray filled area represents the weight overlap area between adjacent rows / columns of convolutional kernels. Detailed Implementation

[0055] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0056] Existing fixed-architecture sensor-memory integrated circuits have rigid computation modes, making it difficult to adapt to the dynamic requirements of different neural networks for convolution kernel size and stride, thus limiting the application potential of sensor-memory integrated vision chips in complex vision tasks. To address these issues, this application provides a reconfigurable convolution weight loading and readout system for a sensor-memory integrated vision chip, thereby resolving the aforementioned problems.

[0057] Figure 1 This is a schematic diagram illustrating the structure of a reconfigurable convolutional weight loading and readout system for a sensor-memory integrated vision chip according to an exemplary embodiment.

[0058] Reference Figure 1 As shown in one embodiment of this application, a reconfigurable convolutional weight loading and readout system 100 for a sensor-memory integrated vision chip includes: a reconfigurable convolutional weight loading system 110, a sensor-memory integrated pixel array 120, a column merging readout system 130, and a timing control module 140.

[0059] The reconfigurable convolution weight loading system 110 is connected to the integrated sensor and computing pixel array 120, the integrated sensor and computing pixel array 120 is connected to the column merging and reading system 130, and the timing control module 140 is connected to the reconfigurable convolution weight loading system 110 and the column merging and reading system 130 respectively.

[0060] The reconfigurable convolution weight loading system 110 is used to configure multi-scale convolution kernels and convert digital weights into analog voltage signals. The analog voltage signals are loaded onto the integrated sensing and computing pixel array 120 using a reconfigurable convolution weight loading mechanism based on a 13×13 weight loading array. The integrated sensing and computing pixel array 120 is used to perform pixel-level visual information perception and parallel convolution calculation on the analog voltage signals. The column merging and reading system 130 is used to read out the parallel convolution calculation results of the integrated sensing and computing pixel array 120. The timing control module 140 is used to provide timing control signals.

[0061] Specifically, the reconfigurable convolutional weight loading system 110 supports programmable convolutional kernel size configurations, such as three standard convolutional kernels: 3×3, 5×5, and 7×7, and supports flexible step size adjustment from 1 to 12 to adapt to the image processing needs of different neural networks in different scenarios. It uses digital-to-analog conversion technology to convert digital weights into analog voltage signals, and dynamically loads the analog voltage signals representing the weights onto the integrated sensing and computing pixel array 120 through a reconfigurable convolutional weight loading mechanism based on a 13×13 weight loading array.

[0062] The 120 pixel array integrates photoelectric sensing and analog computing functions, achieving parallel convolution calculation while completing pixel-level visual perception, and outputting a current signal.

[0063] The column merging readout system 130, based on the programmable column merging switch 131-1 and the integrating capacitor 131-2, supports flexible configuration of multi-scale convolution kernel size, enabling efficient readout of the parallel convolution calculation results of the integrated sensing and computing pixel array 120.

[0064] The timing control module 140 is used to provide timing control signals for the reconfigurable convolution weight loading system 110 and the column merging readout system 130, so as to realize clock synchronization of the reconfigurable convolution weight loading system 110 and the column merging readout system 130, and ensure functional stability during the multi-scale convolution kernel size switching process.

[0065] The reconfigurable convolution weight loading system 110, the integrated pixel array 120, and the column merging and reading system 130 work together through the timing control signals provided by the timing control module 140.

[0066] In the above embodiments of this application, a reconfigurable convolutional weight loading system 110 is used to configure multi-scale convolutional kernels and convert digital weights into analog voltage signals. A reconfigurable convolutional weight loading mechanism based on a 13×13 weight loading array is also used to load the analog voltage signals into the integrated sensor-computer pixel array 120, thereby achieving zero-delay loading of weight signals and improving processing speed. Under the timing control signal provided by the timing module, the column merging readout system 130 can flexibly read out the convolution calculation results of different scale convolutional kernel sizes and strides, thereby realizing dynamic loading and readout of weights, meeting the dynamic requirements of different neural networks for convolutional kernel size and convolutional stride, realizing diversified visual processing and improving hardware resource utilization.

[0067] Reference Figure 1 As shown in some specific embodiments of this application, the reconfigurable convolution weight loading system 110 includes a weight voltage mapping module 111, a weight loading module 112, and a row selection module 113. The weight voltage mapping module 111 is used to convert digital weights into analog voltage signals, the weight loading module 112 is used to dynamically load multi-scale convolution kernels, and the row selection module 113 is used to load analog voltage signals.

[0068] Specifically, the external digital weights are input into the reconfigurable convolutional weight loading system 110, and pass through the weight voltage mapping module 111, the weight loading module 112 and the row selection module 113 in sequence, and are finally input into the integrated sensing and computing pixel array 120.

[0069] Figure 2 This is a schematic diagram of the circuit structure of a weighted voltage mapping module according to an exemplary embodiment.

[0070] Reference Figure 2 As shown, the weighted voltage mapping module 111 includes a digital weight storage unit 111-1, a digital-to-analog converter (DAC) circuit, and a voltage regulation and drive circuit 111-3.

[0071] Specifically, the weighted voltage mapping module 111 adopts a three-level cascaded structure to realize the mapping and loading functions of digital weights.

[0072] The digital weight storage unit 111-1 includes a memory and a register. The memory is used to store the quantized digital weights, and the register is used to store the digital weights to be processed.

[0073] Specifically, the memory can be SRAM, eFlash, or RRAM.

[0074] For example, the memory stores the quantized input weights, such as 8-bit fixed-point numbers.

[0075] The digital-to-analog converter circuit 111-2 is used to convert digital weights into analog voltage signals.

[0076] Specifically, the digital-to-analog converter circuit 111-2 can be a resistive digital-to-analog converter (R-DAC), a current-driven digital-to-analog converter, or a pulse-width modulation digital-to-analog converter (PWM-DAC).

[0077] Among them, resistive digital-to-analog converters can use adjustable resistor networks, such as switched capacitor arrays, to generate analog voltage signals corresponding to digital weights.

[0078] The current-steering digital-to-analog converter is used to perform high-precision mapping of digital weights to generate analog voltage signals.

[0079] Pulse width modulation (PWM) digital-to-analog converters (DPCs) map digital weights to analog voltage signals by modulating the voltage with a duty cycle. PWM PCs can be applied to low-power scenarios.

[0080] The voltage regulation and drive circuit 111-3 includes a noise suppression circuit, a programmable voltage bias circuit, and an operational amplifier. The noise suppression circuit is used to perform correlated double sampling, the programmable voltage bias circuit is used to dynamically adjust the reference voltage or bias point, and the operational amplifier is used to amplify the input signal to a preset level.

[0081] Specifically, the noise suppression circuit reduces the impact of thermal noise through Correlated Double Sampling (CDS).

[0082] Programmable voltage bias circuits adapt to different weighting scales by dynamically adjusting the reference voltage or bias point.

[0083] Operational amplifiers enhance driving capability and ensure voltage stability by amplifying the input signal to a preset level.

[0084] For example, the digital weights are first stored in the register of the digital weight unit, then quantized and stored in memory, such as SRAM or eFlash memory. Then, via the digital-to-analog converter circuit 111-2, the digital weights are mapped into high-precision analog voltage signals through a programmable digital-to-analog converter, such as a resistive digital-to-analog converter. The weight voltage is then dynamically adjusted by the voltage regulation and drive circuit 111-3 to ensure that the weight voltage can be dynamically adjusted within a preset voltage range, while improving the stability and anti-interference capability of the weight voltage output.

[0085] Figure 3This is a schematic diagram illustrating the operation of a weight loading module according to an exemplary embodiment.

[0086] Reference Figure 3 As shown, the weight loading module 112 is used to arrange multi-scale convolution kernels on a preset basic weight loading array using a preset weight arrangement method, and uses the preset basic weight loading array as the basic building unit to construct the integrated sensing and computing pixel array 120 using a tile-type expansion strategy. The row weights and column weights of the preset basic weight loading array have some overlapping areas. The preset basic weight loading array uses a reconfigurable convolution weight loading mechanism based on a 13×13 weight loading array to load the analog voltage signal onto the integrated sensing and computing pixel array 120.

[0087] Specifically, the weight loading module 112 adopts a hierarchical dynamic weight loading structure and achieves zero-delay loading of weight signals through global parallel connection technology. The preset basic weight loading array adopts a 13×13 basic weight loading array, and the weight loading module 112 adopts a 13×13 weight loading array 112-1.

[0088] The analog voltage signal output by the weighted voltage mapping module 111 can be the analog voltage signal of the weighted voltage of 3×3, 5×5, and 7×7 convolutional kernels. According to the target convolutional kernel size, the 3×3, 5×5, and 7×7 convolutional kernels are arranged in a preset weighted arrangement in the 13×13 weighted loading array 112-1. An optimized tile-style expansion strategy is adopted, and the 13×13 weighted loading array 112-1 is used as the basic building block to maximize hardware utilization efficiency on the M×N integrated sensor and computing pixel array 120.

[0089] Among them, the row / column weights between adjacent 13×13 weight loading arrays 112-1 have overlapping areas. The overlapping areas adopt a bidirectional connection mechanism, so that the shared boundary pixels of the overlapping areas can receive weight inputs from the left and right or up and down.

[0090] Each 13×13 weight loading array 112-1 supports dynamic configuration of multi-scale convolutional kernels. Each 13×13 weight loading array 112-1 can be dynamically configured with 6×6 3×3 convolutional kernels, 3×3 5×5 convolutional kernels, or 2×2 7×7 convolutional kernels, with adjacent convolutional kernels maintaining a row / column overlap. The preset weight arrangement is as follows:

[0091] Figure 4 This is a schematic diagram illustrating the connection rule of a 3×3 convolution kernel in a 13×13 weighted array according to an exemplary embodiment.

[0092] Reference Figure 4As shown, for 3×3 convolutional kernels, an overlapping tile arrangement strategy is adopted to map 6×6 3×3 convolutional kernels to a 13×13 array in a sliding window manner. Adjacent convolutional kernels maintain an overlapping area of ​​one row or one column, thereby achieving seamless coverage of the input feature map.

[0093] The details are as follows: Figure 4 In the middle (a), it represents a 3×3 convolution kernel. Figure 4 In Figure (b), the spatial distribution of 6×6 3×3 convolutional kernels in the 13×13 weight loading array 112-1 is shown. The gray filled area represents the weight overlap area between adjacent rows / columns of convolutional kernels. The 3×3 convolutional kernels are arranged in the 13×13 weight loading array 112-1 using an optimized 6×6 overlapping tile arrangement to arrange 36 3×3 convolutional kernels in the 13×13 weight loading array 112-1 with a sliding stride of 2, forming an overlapping area of ​​one row / column weight between adjacent convolutional kernels, thus achieving seamless coverage of the feature map.

[0094] Figure 5 This is a schematic diagram illustrating the connection rules in a 13×13 weighted array after a 3×3 convolution kernel slides to the right once, according to an exemplary embodiment.

[0095] Reference Figure 5 As shown, Figure 5 In the middle (a), the kernel is represented after the 3×3 kernel has been slid to the right once. Figure 5 In the middle (b), the spatial distribution of 6×6 3×3 convolution kernels after sliding to the right once is shown in the 13×13 weight loading array 112-1. The gray filled area represents the weight overlap area between adjacent rows / columns of convolution kernels.

[0096] Reference Figure 4 , Figure 5 As shown, this embodiment demonstrates the connection state after the 3×3 convolution kernel is shifted to the right with a sliding step of 1, reflecting the flexible feature of the 13×13 weighted array 112-1 supporting a programmable sliding step of 1-12.

[0097] Figure 6 This is a schematic diagram illustrating the connection rule of a 5×5 convolution kernel in a 13×13 weighted array according to an exemplary embodiment.

[0098] Reference Figure 6 As shown, for 5×5 convolutional kernels, an overlapping tile arrangement strategy is adopted, which maps 3×3 5×5 convolutional kernels to a 13×13 weight loading array 112-1 in a sliding window manner. Adjacent convolutional kernels maintain an overlapping area of ​​one row or one column, thereby achieving seamless coverage of the input feature map.

[0099] Specifically, Figure 6 In the middle (a), it represents a 5×5 convolution kernel. Figure 6 In Figure (b), the spatial distribution of 3×3 5×5 convolutional kernels in the 13×13 weight loading array 112-1 is shown. The gray filled area represents the weight overlap area between adjacent rows / columns of convolutional kernels. It adopts a 3×3 overlapping tile arrangement. The nine 5×5 convolutional kernels are arranged in the 13×13 weight loading array 112-1 with a stride of 4, and the overlap of the weights of adjacent rows / columns between convolutional kernels is maintained.

[0100] Figure 7 This is a schematic diagram illustrating the connection rules of a 7×7 convolution kernel in a 13×13 weighted array according to an exemplary embodiment.

[0101] Reference Figure 7 As shown, for 7×7 convolutional kernels, an overlapping tile arrangement strategy is adopted, which maps 2×2 7×7 convolutional kernels to a 13×13 weight loading array 112-1 in a sliding window manner. Adjacent convolutional kernels maintain an overlapping area of ​​one row or one column, thereby achieving seamless coverage of the input feature map.

[0102] Specifically, Figure 7 In the middle (a), it represents a 7×7 convolution kernel. Figure 7 In Figure (b), the spatial distribution of 2×2 7×7 convolutional kernels in the 13×13 weight loading array 112-1 is shown. The gray filled area represents the weight overlap area between adjacent rows / columns of convolutional kernels. It adopts a 2×2 overlapping tile arrangement. The four 7×7 convolutional kernels are arranged in the 13×13 weight loading array 112-1 with a stride of 6, and the overlap of the weights of adjacent rows / columns between convolutional kernels is maintained.

[0103] The above embodiments of this application, by adopting a preset weight arrangement, realize multi-scale convolution kernel configuration of 3×3 convolution kernel, 5×5 convolution kernel, and 7×7 convolution kernel, maintain a fixed overlap factor, and realize that convolution kernels of different sizes can fully utilize the hardware resources of the 13×13 weight loading array 112-1, and load the weight data into the M×N sensor-computer integrated pixel array 120 in a reconfigurable manner.

[0104] In the weight loading module 112, when an analog voltage signal of any weight voltage, such as W1, is input to any coordinate position of the 13×13 weight loading array 112-1, such as the <1,1> position, since the corresponding positions of all 13×13 weight loading arrays 112-1 in the M×N integrated pixel array 120 share the same physical connection network, the analog voltage signal W1 is synchronously loaded to the <1,1> position of all 13×13 weight loading arrays 112-1 that make up the M×N integrated pixel array 120, thereby achieving zero-delay global weight loading.

[0105] In the above embodiments of this application, the analog voltage signal of digital weight is sequentially loaded into the weight input port corresponding to each integrated sensor pixel 120-1 in the M×N integrated sensor-computer pixel array 120 through the global parallel connection structure of the weight loading module 112.

[0106] Figure 8 This is a schematic diagram of the circuit structure of a row selection module according to an exemplary embodiment.

[0107] Reference Figure 8 As shown, the row selection module 113 includes M row selection switches 113-1, each row selection switch 113-1 corresponding to control each row of pixels 120-1 of the integrated sensor and computing pixel array 120.

[0108] Specifically, the M row selection switches 113-1 correspond to the M rows of integrated sensing and computing pixel arrays 120, and each row selection switch 113-1 corresponds to one row of integrated sensing and computing pixel arrays 120. The i-th row selection switch 113-1 receives the row selection signal RSEL corresponding to the i-th row pixel 120-1. Control.

[0109] Figure 9 This is a schematic diagram of the circuit structure of a row selection switch according to an exemplary embodiment.

[0110] Reference Figure 9 As shown, the row selection switch 113-1 includes MOSFETs M0 to M1. 25 And inverter I0, MOSFETs M0~M 12 The gates are uniformly connected to the row select signal RSEL of the i-th row. Its MOS transistors M0 to M 12 The source and drain are connected to 13 different weight input and output ports respectively to adapt to the side length of the 13×13 weight loading array 112-1, that is, the minimum number of non-repeating weights corresponding to each row of pixels 120-1 in the integrated sensing and computing pixel array 120.

[0111] The input port of inverter I0 is connected to the row selection signal RSEL. The output port is connected to the MOSFET M 13 ~M 25 The gate of the MOSFET is connected to the gate of the MOSFET. 13 ~M 25 The source and drain are connected to ground and 13 different output ports, respectively.

[0112] For example, when it is necessary to select the i-th row pixel 120-1, the row selection signal RSEL Set to high level, row selection switch When the signal is turned on, the analog voltage signal of the weighted voltage is directly transmitted to the i-th row of the integrated sensor-computer pixel array 120. Since the weight loading module 112 transmits the analog voltage signal through a physical interconnect network, and the side length of the 13×13 weight loading array 112-1 is 13, only 13 analog voltage signals need to be directly input, corresponding to the 13 inputs / outputs of the row selection switch 113-1. The remaining analog voltage signals are synchronously loaded into the selected row of pixels 120-1, improving signal transmission efficiency. When it is necessary to turn off the i-th row of pixels 120-1, the row selection signal RSEL... Keep the level low, at which point the row selection switch... Turn off, forcing the output of the analog voltage signal to 0V.

[0113] In this embodiment, the analog voltage signal output by the weight loading module 112 is efficiently input to the corresponding pixel 120-1 of the sensing and computing integrated pixel array 120 after being selected and controlled by the row selection switch 113-1 of the row selection module 113.

[0114] The selection mechanism of the row selection switch 113-1 is controlled by the timing control signal provided by the timing control module 140. The row selection switch 113-1 is dynamically turned on according to the convolution kernel size. If the convolution kernel size is 3×3, then 3 rows of pixels 120-1 are turned on. If the convolution kernel size is 5×5, then 5 rows of pixels 120-1 are turned on. If the convolution kernel size is 7×7, then 7 rows of pixels 120-1 are turned on, thereby realizing reconfigurable convolution. At the same time, by turning off the non-working row pixel 120-1 circuit, the static power consumption is reduced.

[0115] In the above embodiments of this application, analog voltage signals are quickly loaded and static power consumption is reduced by using the row selection switch 113-1 and the reconfigurable convolution weight loading mechanism based on the 13×13 weight loading array.

[0116] In some specific embodiments of this application, the integrated pixel array 120 for sensing and computing adopts a CMOS process integration scheme or a heterogeneous integration scheme.

[0117] For example, the silicon-based integrated sensing and computing pixel unit can be integrated using standard CMOS technology. Each pixel 120-1 integrates a photodiode, analog computing circuitry, and has a weighted input port and a current-mode output port. In particular, the output ports of each column of pixel units in the integrated sensing and computing pixel array 120 are connected to a shared column bus through an optimized interconnect structure.

[0118] For example, for non-silicon-based integrated sensing and computing pixel units, including infrared detectors (InGaAs material) or perovskite-based integrated sensing and computing pixel units, optocoupler is achieved through direct injection (DI). The non-silicon-based integrated sensing and computing pixel array 120 is three-dimensionally integrated with a reconfigurable convolutional weight loading and readout system through vertical stacking or heterogeneous integration processes.

[0119] For infrared applications, InGaAs materials or type II superlattice infrared detectors can be three-dimensionally integrated with silicon-based processing circuits using through-silicon via (TSV) technology. High-efficiency optocouplers are achieved through direct injection (DI) and array configurations of any size (M×N) can be supported to meet high-resolution requirements.

[0120] The above embodiments of this application provide a CMOS process-compatible integrated pixel array 120 that is compatible with non-silicon-based pixel arrays. For non-silicon-based pixel arrays, the application prospects can be expanded through three-dimensional integration technology, providing technical support for the development of new integrated visual chips.

[0121] Figure 10 This is a schematic diagram illustrating the collaborative operation of a sensor-computer integrated pixel array and column merging readout system according to an exemplary embodiment.

[0122] Reference Figure 10 As shown, the column merging and readout system 130 includes a column merging module 131, a sample / hold module 132, and a multiplexing module 133. The column merging module 131 is used to merge current signals according to the reconfigurable convolution kernel size and read out analog voltage signals. The sample / hold module 132 is used to transmit analog voltage signals to the multiplexing module 133. The multiplexing module 133 is used to switch signal paths.

[0123] Among them, the current signal output from the integrated pixel array 120 is input to the column merging module 131, the sampling / holding module 132, and the multiplexing module 133, and is finally output to the reconfigurable convolution weight loading and readout system of the integrated vision chip.

[0124] Specifically, the column merging module 131 includes N-1 column merging switches 131-1 <1> - <n-1>and N sets of integrating capacitors 131-2 <1> - <n>Each set of integrating capacitors 131-2 has its input terminal connected to the column line of the integrated inductive and computational pixel array 120, and each switch is connected to the adjacent column line of the integrated inductive and computational pixel array 120.

[0125] Where N corresponds to the total number of columns in the integrated sensor-computer pixel array 120.

[0126] The sampling / holding module 132 includes N sets of sampling / holding circuits 132-1, each set of sampling / holding circuits 132-1 being coupled to each set of integrating capacitors 131-2.

[0127] Each sample-and-hold circuit 132-1 receives the analog voltage signal from each integrating capacitor 131-2 as an input and outputs it to the multiplexing module 133.

[0128] The multiplexing module 133 includes multiplexers, and the input of each multiplexer is connected to a preset number of sample / hold circuits 132-1.

[0129] In this application, the multiplexing module 133 includes Group 6:1 multiplexer. Among them, the multiplexer... <1> The input terminal is connected to the sample-and-hold circuit 132-1 <1> ~ <6> Multiplexer <t>Connect the input terminals to the sample / hold circuits 132-1<6t-5>~<6t>, until all N columns of the sample / hold circuits 132-1 are connected.

[0130] Reference Figure 10 As shown, the M×N integrated pixel array 120 realizes efficient reconfigurable parallel convolution operation. The output of each pixel unit is connected to the column line to form a column parallel signal and form a transmission path in the column merging module 131 and the sample / hold module 132.

[0131] When the reconfigurable convolution weight loading and readout system 100 of the integrated sensor-memory vision chip is in operation, the row selection module 113 dynamically selects the corresponding k rows of pixels 120-1 according to the current convolution kernel size, i.e., 3×3, 5×5, or 7×7, where k = 3, 5, or 7. Based on the current-mode operation mechanism, the output current of the selected pixel 120-1 follows Kirchhoff's current law, achieving current domain accumulation on the column lines, thereby cooperating with pixel 120-1 to achieve a complete convolution multiplication-accumulation operation.

[0132] The column merging module 131 includes N-1 column merging switches 131-1 and N sets of integrating capacitors 131-2, where N is the total number of columns in the array. The column merging module 131 dynamically adjusts the column merging strategy according to the convolution kernel size: for a 3×3 convolution kernel, the 3 columns of signals are merged into the same integrating capacitor 131-2; for a 5×5 or 7×7 convolution kernel, the 5 columns or 7 columns of signals are merged into the same integrating capacitor 131-2 respectively, thereby effectively supporting analog domain multiplication and accumulation operations of convolution kernels of different sizes through a reconfigurable architecture.

[0133] The sample / hold module 132 contains N sets of sample / hold circuits 132-1, each circuit being coupled to a corresponding integrating capacitor 131-2. All sampling results are output to the multiplexing module 133.

[0134] Figure 11 This is a schematic diagram illustrating the collaborative operation of a sample / hold module and a multiplexing module according to an exemplary embodiment.

[0135] Reference Figure 11 As shown, the multiplexing module 133 adopts a grouped parallel output architecture, specifically including... The system employs a 6:1 multiplexer 133-1 with rounding up. This design leverages the system's ability to support up to 7×7 convolutional kernels with adjacent kernels maintaining a 1-row / column overlap. Each multiplexer 133-1 precisely corresponds to the output of the 6 columns of sample-and-hold circuits 132-1 in the sample-and-hold module 132. This architecture achieves an optimal balance between hardware resource utilization and readout efficiency while ensuring signal throughput.

[0136] The timing control module 140 of this application is used to provide clock synchronization timing control signals for the reconfigurable convolution weight loading system 110 and the column merging readout system 130. Furthermore, according to the convolution kernels of different scales, the gating logic of the row selection switch 113-1 of the row selection module 113 in the reconfigurable convolution weight loading system 110 and the column merging switch 131-1 of the column merging module 131 in the column merging readout system 130 is dynamically adjusted to realize the reconfigurable convolution function.

[0137] This application provides a reconfigurable convolutional weight loading and readout system 100 for a sensor-memory integrated vision chip. Through innovative architecture design and circuit optimization, it supports the dynamic loading of various standard convolutional kernel sizes, such as 3×3, 5×5, and 7×7 convolutional kernels, improving the flexibility of the vision processing system and its adaptability to different neural network models. Furthermore, this system is not only suitable for traditional silicon-based sensor-memory integrated computing circuits but can also be integrated in three dimensions with non-silicon-based infrared detectors, thus possessing advantages in compatibility and scalability. This broad material compatibility enables the technical solution to meet pixel-level reconfigurable vision processing requirements from the visible light to the infrared band, providing key technical support for the development of next-generation intelligent vision chips.

[0138] The specific embodiments of this application have been described above. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of this specification, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.< / t> < / n>

Claims

1. A reconfigurable convolutional weight loading and readout system for an integrated sensing and computing vision chip, characterized in that, include: Reconfigurable convolutional weight loading system, integrated sensing and computing pixel array, column merging and readout system, and timing control module; The reconfigurable convolution weight loading system is connected to the integrated sensing and computing pixel array, and the integrated sensing and computing pixel array is connected to the column merging and readout system. The timing control module is connected to both the reconfigurable convolution weight loading system and the column merging and readout system. The reconfigurable convolution weight loading system is used to configure multi-scale convolution kernels and convert digital weights into analog voltage signals. The analog voltage signals are loaded onto the integrated sensing and computing pixel array using a reconfigurable convolution weight loading mechanism based on a 13×13 weight loading array. The integrated sensing and computing pixel array is used to perform pixel-level visual information perception and parallel convolution calculation on the analog voltage signals. The column merging and readout system is used to read out the parallel convolution calculation results of the integrated sensing and computing pixel array. The timing control module is used to provide timing control signals.

2. The reconfigurable convolutional weight loading and readout system for the integrated sensing and computing vision chip according to claim 1, characterized in that, The reconfigurable convolution weight loading system includes a weight voltage mapping module, a weight loading module, and a row selection module. The weight voltage mapping module is used to convert the digital weights into the analog voltage signal. The weight loading module is used to dynamically load multi-scale convolution kernels. The row selection module is used to load the analog voltage signal.

3. The reconfigurable convolutional weight loading and readout system for the integrated sensing and computing vision chip according to claim 2, characterized in that, The weighted voltage mapping module includes a digital weight storage unit, a digital-to-analog converter circuit, and a voltage regulation and drive circuit. The digital weight storage unit includes a memory and a register. The memory is used to store quantized digital weights, and the register is used to store digital weights to be processed. The digital-to-analog converter circuit is used to convert the digital weights into the analog voltage signal; The voltage regulation and drive circuit includes a noise suppression circuit, a programmable voltage bias circuit, and an operational amplifier. The noise suppression circuit is used for correlated double sampling, the programmable voltage bias circuit is used for dynamically adjusting the reference voltage or bias point, and the operational amplifier is used to amplify the input signal to a preset level.

4. The reconfigurable convolutional weight loading and readout system for the integrated sensing and computing vision chip according to claim 2, characterized in that, The weight loading module is used to arrange multi-scale convolutional kernels on a preset basic weight loading array using a preset weight arrangement method, and to construct an M×N integrated sensor-computer pixel array using the preset basic weight loading array as the basic building unit and a tile-type expansion strategy. The row weights and column weights of the preset basic weight loading array have some overlapping areas. The preset basic weight loading array uses a reconfigurable convolutional weight loading mechanism based on a 13×13 weight loading array to load the analog voltage signal onto the M×N integrated sensor-computer pixel array.

5. The reconfigurable convolutional weight loading and readout system for the integrated sensing and computing vision chip according to claim 2, characterized in that, The row selection module includes M row selection switches, each row selection switch corresponding to control each row of pixels in the M×N integrated sensor and computing pixel array.

6. The reconfigurable convolutional weight loading and readout system for the integrated sensing and computing vision chip according to claim 1, characterized in that, The column merging and readout system includes a column merging module, a sample / hold module, and a multiplexing module. The column merging module is used to merge current signals according to the reconfigurable convolution kernel size and read out the analog voltage signal. The sample / hold module is used to transmit the analog voltage signal to the multiplexing module, and the multiplexing module is used to switch signal paths.

7. The reconfigurable convolutional weight loading and readout system for the integrated sensing and computing vision chip according to claim 6, characterized in that, The column merging module includes N-1 column merging switches and N sets of integrating capacitors. The input terminal of each set of integrating capacitors is connected to the column line of the integrated pixel array, and each switch is connected to the adjacent column line of the integrated pixel array.

8. The reconfigurable convolutional weight loading and readout system for the integrated sensing and computing vision chip according to claim 7, characterized in that, The sampling / holding module includes N sets of sampling / holding circuits, each set of sampling / holding circuits being coupled to each set of integrating capacitors.

9. The reconfigurable convolutional weight loading and readout system for the integrated sensing and computing vision chip according to claim 8, characterized in that, The multiplexing module includes a multiplexer, and the input of each multiplexer is connected to a preset number of sample / hold circuits.

10. The reconfigurable convolutional weight loading and readout system for the integrated sensing and computing vision chip according to claim 1, characterized in that, The integrated pixel array for sensing and computing adopts a CMOS process integration scheme or a heterogeneous integration scheme.

Images