Image recognition system based on FPGA-driven inductive focal plane and memristor

Abstract

This invention discloses an image recognition system based on FPGA-driven sensing focal plane array and memristor, comprising a photoelectric sensing and front-end convolutional detector array, an analog computing memristor array, a signal driving and path control unit, and an analog computing readout and sampling processing unit. The system architecture utilizes a unified FPGA scheduling + heterogeneous hardware to share computational tasks, enabling the most computationally intensive convolutional and fully connected layers in the neural network to be accelerated by the most suitable external hardware. This fully leverages the high parallel sensing advantage of the photoelectric sensing array in the first convolutional layer and the high energy efficiency of the memristor matrix in large-scale matrix operations in the analog domain. Simultaneously, by reusing the driving circuit and signal processing link, this invention significantly reduces the system's hardware resource consumption and energy usage, achieving high-performance, low-power intelligent image recognition in edge computing scenarios.

Description

Technical Field

[0001] This invention belongs to the field of image processing and hardware acceleration technology, specifically relating to an image recognition system based on FPGA-driven sensing focal plane and memristor. Background Technology

[0002] With the rapid development of artificial intelligence technology, image recognition has been widely used in security monitoring, infrared detection, gesture recognition, intelligent interaction and other scenarios; especially in edge computing devices, the system has put forward higher requirements for the processing speed, integration and energy efficiency of image recognition algorithms.

[0003] To address these needs, a computing architecture called In-Sensor Computing has been proposed to move image perception and neural network computation forward to the image acquisition end. In this architecture, the image detector not only performs traditional image acquisition functions but also integrates basic computing units (such as convolution kernel weight circuits) to perform the convolution operation of the first layer of the neural network on the focal plane. This design can significantly reduce the front-end data bandwidth requirements, reduce system latency and power consumption, and enhance support for real-time processing.

[0004] In such systems, FPGAs (Field-Programmable Gate Arrays) typically serve as back-end computing modules, undertaking neural network inference tasks after the first convolutional layer, such as intermediate convolutional layers, fully connected layers, and pooling layers. Due to their high parallelism and programmability, FPGAs have become important computing platforms in sensing focal plane array systems. A typical structure includes an image detector array, a photoelectric signal amplification module (TIA), an analog-to-digital converter (ADC), and the FPGA main control unit.

[0005] To further improve the computational performance of neural networks, researchers have proposed using novel devices such as memristors in the inference stage of neural networks. Memristors can perform voltage-driven vector-matrix multiplication calculations in a two-dimensional array and also have non-volatile storage capabilities, thus they are often used to accelerate computationally intensive modules such as convolutional layers or fully connected layers. The typical advantages of memristors are low power consumption, simple array structure, and certain analog parallel computing capabilities, making them suitable for resource-constrained edge scenarios. For example, Chinese patent technology with publication number CN112101549B provides a neural network training and inference device based on a memristor array. By adjusting the conductance of the memristor in stages with pulses, the update of neural network weight parameters is optimized, while ensuring training accuracy and speed. This device can be directly applied to the inference stage, utilizing the parallel computing capabilities of the memristor array to accelerate matrix multiplication.

[0006] However, in existing designs, the inductive focal plane array system and the memristor acceleration module are usually built as separate processing units. These architectures generally have the following characteristics:

[0007] 1. The two computing systems are each configured with independent ADC / DAC modules;

[0008] 2. Both are driven by independent control logic;

[0009] 3. Data needs to be transmitted across multiple modules, resulting in complex system communication links;

[0010] 4. The various parts lack a unified data interface and resource sharing mechanism.

[0011] Furthermore, since the inductive focal plane and memristor are based on analog signal processing links, their circuit structures often involve DACs (digital-to-analog converters) for weight application and ADCs for output sampling. In existing solutions, the analog-to-digital conversion resources used by the two subsystems are often not reused, resulting in redundant configuration of the ADC / DAC modules in the overall system, low resource utilization efficiency, and high pressure on circuit area and power consumption.

[0012] Therefore, the typical design of current image recognition systems presents a distributed structure of perception-computation-storage, lacking a compact system architecture that can achieve unified control, resource sharing, and path integration at the hardware level. Summary of the Invention

[0013] In view of the above, the present invention provides an image recognition system based on FPGA-driven sensing focal plane and memristor. This system organically combines photoelectric detection, analog computing and digital reasoning to form a hardware architecture that integrates sensing, storage computing and reasoning. It also achieves unified control of the sensing detector and memristor through the reuse of driving circuits, so as to realize the rapid acquisition and efficient recognition of image information.

[0014] An image recognition system based on FPGA-driven sensing focal plane and memristor includes:

[0015] The photoelectric sensing and front-end convolutional detection array acquires image data by sensing light intensity and adjusts the output photocurrent signal according to the applied bias voltage to achieve linear adjustment of light intensity and bias voltage. Thus, the pixel convolution calculation of light intensity × bias voltage, i.e. the first layer convolution operation, is directly completed at the hardware level.

[0016] Analog computational memristor arrays are used to implement matrix multiplication and addition operations in convolutional layers and fully connected layers other than the first convolutional layer during neural network inference.

[0017] Signal drive and path control unit is used to drive photoelectric sensing and front-end convolutional detection array or analog computing memristor array. The FPGA is used to schedule and select the corresponding pixels and memristors to participate in the operation.

[0018] The analog computing readout and sampling processing unit is used to process the output of the photoelectric sensing and front-end convolutional detector array or the analog computing memristor array, and then use the FPGA to perform non-convolutional operations in the neural network inference process in the digital domain to complete the image recognition and classification task.

[0019] Furthermore, the photoelectric sensing and front-end convolutional detector array and the analog computing memristor array share the same set of signal driving and path control unit and analog computing readout and sampling processing unit.

[0020] Furthermore, the photoelectric sensing and front-end convolutional detection array is composed of multiple pixels arranged in an array. The pixels sense the illumination intensity of incident light in different wavelength bands and adjust the output of the photocurrent signal according to the applied bias voltage. The photocurrent signal is linearly related to the illumination intensity and the bias voltage.

[0021] Furthermore, the conductance values ​​of the memristors in the analog computational memristor array are pre-written into the corresponding neural network weights. The applied input voltage vector generates an output current vector through the memristor array, which is the matrix operation result.

[0022] Furthermore, the signal driving and path control unit includes a voltage output module, a parallel control module, and a path switching module. The voltage output module employs a DAC array and is controlled by the parallel control module to provide bias voltage to the pixels or input voltage to the memristors. The path switching module employs an analog switch array and is controlled by the parallel control module to select the appropriate pixels or memristors for computation. The parallel control module uses an FPGA to uniformly manage the timing and data flow of the pixels and memristors through scheduling. Under FPGA control, the voltage output module and path switching module of this unit achieve unified driving of the sensor and memristors. This allows for providing row selection voltage and photosensitive bias voltage to the sensor to perform the first layer of convolution calculation, and also provides the multi-port drive voltage required for programming and reading the memristors to perform subsequent convolution and fully connected operations, thereby reducing redundant hardware and improving resource utilization.

[0023] Furthermore, the analog computation readout and sampling processing unit includes a signal amplification and pre-sampling buffer module, an analog signal filtering module, an analog signal sampling module, a path switching module, and a neural network inference module. The path switching module uses an analog switch array and is controlled by the neural network inference module, selecting the output of the corresponding memristor for subsequent processing. The signal amplification and pre-sampling buffer module uses a TIA (transimpedance amplifier) ​​array to amplify the output of the photoelectric sensing and front-end convolutional detection array or the analog computation memristor array. The analog signal filtering module uses an active RC (resistor-capacitor) array to filter the amplified output. The analog signal sampling module uses an ADC array to convert the filtered output into a digital signal. The neural network inference module uses an FPGA, which performs non-convolution operations (summing, pooling, normalization, nonlinear activation, etc.) on the digital signal in the digital domain and outputs the results to complete the image recognition and classification task. The analog signal link of this unit can also be reused for the output signal processing of the detector and memristor: when the detector is working, its output photoresponse convolution current is amplified, filtered and sampled; when the memristor is working, its output analog calculation result current is processed in the same way, and the digitized data is input to the FPGA for subsequent digital domain operations and result output.

[0024] Furthermore, the parallel control module and the neural network inference module share the same FPGA.

[0025] Furthermore, the workflow of the image recognition system is as follows:

[0026] (1) According to the neural network weight values ​​to be loaded, the FPGA controls the path switching module and the voltage output module to apply the write voltage to the rows, columns and selection terminals of the analog computing memristor matrix in sequence, so as to complete the weight rewriting and calibration of the memristor array.

[0027] (2) The FPGA switches the drive to the photoelectric sensing and front-end convolution detection array through the path switching module, and uses the voltage output module to output the gate voltage to activate the pixels in the corresponding row of the array. While the row is activated, the FPGA controls another voltage output module to apply the bias voltage of the corresponding column mapping of the convolution kernel in sequence, so that the pixel physically realizes the pixel convolution operation of light intensity × bias voltage. The pixel generates photocurrent linearly according to the incident light intensity and the applied bias voltage and completes the initial accumulation in the array according to the row and column, and outputs the analog photocurrent signal representing the corresponding convolution window. Then the FPGA controls the signal amplification and sampling pre-buffer module, the analog signal filtering module and the analog signal sampling module to perform current-to-voltage conversion, amplification, filtering and analog-to-digital conversion on the analog photocurrent signal to complete the analog signal sampling of the convolution window. The FPGA slides the convolution window in the vertical or horizontal direction according to the convolution kernel stepping strategy, and repeats the above three-step operation of row selection, column bias and sampling to obtain the convolution results of different convolution windows. Finally, the FPGA sums the output of each window according to the corresponding position in the digital domain to obtain the complete feature map.

[0028] (3) The FPGA switches the drive to the analog computing memristor array through the path switching module. The voltage output module applies the input voltage vector to the array, triggering the memristor array to generate an analog output of matrix multiplication and addition. The analog output of the memristor array is also sent to the FPGA after being amplified, filtered and converted by the signal amplification and sampling pre-buffering module, the analog signal filtering module and the analog signal sampling module. The FPGA completes the interlayer nonlinear processing, including pooling, normalization and activation, in the digital domain to obtain the image recognition and classification results and send them to the host computer for display and recording.

[0029] This invention aims to solve the following technical problems existing in the prior art:

[0030] 1. To address the issue of redundant hardware resource configuration caused by the separate deployment of existing inductive focal plane array systems and memristor computing modules, this invention aims to propose a structure that enables the reuse of analog-to-digital conversion resources (such as ADC and DAC), thereby reducing system area and power consumption overhead and improving hardware resource utilization efficiency.

[0031] 2. Given that existing image recognition systems use independent data paths for sensing and memristor computing modules and have inconsistent interface standards, resulting in frequent data flow conversions across modules and complex communication control, the purpose of this invention is to unify the data path and standardize the interface in the sensing computing and memristor computing processes, thereby simplifying the system structure, reducing communication latency, and improving overall computing efficiency.

[0032] 3. In existing systems, the control logic is decentralized, with the inductive computing and memristor modules each relying on independent control mechanisms, lacking collaborative management capabilities and making it difficult to achieve centralized scheduling and task allocation. Another objective of this invention is to improve system integration and task adaptability by introducing a unified programmable control platform to achieve centralized coordination between the inductive computing and memristor computing modules.

[0033] In summary, this invention aims to address the technical problems existing in current image recognition computing systems, such as resource redundancy, inconsistent data interfaces, and weak inter-module collaboration capabilities. It constructs an intelligent image recognition system architecture suitable for edge sensing nodes, featuring a compact structure, integrated computing paths, and efficient resource reuse. This invention unifies photoelectric sensing detectors, analog memristors, and FPGAs into a single integrated sensing-memory-computing system. Furthermore, it achieves hardware resource sharing through the multiplexing of driver and signal links such as DACs / analog switches and TIAs / ADCs. Therefore, this invention has the following beneficial technical effects:

[0034] 1. Reduced data movement and improved real-time performance. This invention moves the first convolutional layer of the neural network to the detector layer (sensor layer) and performs subsequent convolutional / fully connected calculations in the analog domain using memristors when needed. This avoids a large number of original pixels being transmitted back and forth on the digital bus, reducing the data movement path from an architectural perspective. This helps to reduce end-to-end recognition latency and improve the system's real-time response capability.

[0035] 2. Improve computational energy efficiency and adaptability of task division. This invention adopts heterogeneous hardware collaboration (the detector performs the first layer of convolution, the memristor performs large-scale matrix multiplication and addition, and the FPGA performs control and digital domain nonlinear operations), so that each type of hardware can undertake the computational tasks it is good at, thereby improving energy efficiency and efficiency of large-scale convolutional computation from the perspective of system architecture.

[0036] 3. Reduce hardware redundancy and system cost. This invention avoids equipping detectors and memristors with separate drive / sampling circuits by reusing DAC arrays, path switching modules, and analog signal links, thereby reducing the number of components and PCB (printed circuit board) area, improving system integration, and reducing manufacturing costs and maintenance complexity.

[0037] 4. Improved system adaptability and scalability. In this invention, the FPGA serves as a unified scheduling and orchestration unit, supporting dynamic allocation of computational tasks under different neural network topologies (e.g., driving memristors when encountering convolution or fully connected computations). This allows the architecture to adapt to various network structures and sensing device types, facilitating engineering deployment and subsequent expansion.

[0038] 5. Enhanced application value in edge scenarios. Due to the aforementioned architectural improvements in latency, power consumption, and cost, this invention is particularly suitable for edge deployment scenarios (such as embedded monitoring, robotics, and low-power terminals) that are sensitive to real-time performance, energy consumption, and cost, and has significant application value and market potential. Attached Figure Description

[0039] Figure 1 This is a block diagram of the intelligent image recognition system based on FPGA-driven sensing focal plane and memristor according to the present invention.

[0040] Figure 2 This is a schematic diagram of the workflow of the intelligent image recognition system of the present invention.

[0041] Figure 3 This is a schematic diagram of the photoelectric sensing and front-end convolutional detector driving architecture in an embodiment of the present invention.

[0042] Figure 4 This is a schematic diagram of the analog computational memristor matrix drive architecture in an embodiment of the present invention. Detailed Implementation

[0043] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Example 1

[0045] This embodiment provides an intelligent image recognition system based on FPGA-driven sensing focal plane array and memristor, comprising two main parts: a signal driving and path control unit and an analog computing readout and sampling processing unit. Using the FPGA as the core scheduling and control unit, it achieves unified driving and signal processing multiplexing of the sensing detector and memristor, enabling efficient collaboration between the first layer of convolution calculation and subsequent convolution and fully connected layer calculations in the neural network. The system as a whole consists of two functional units: the signal driving and path control unit and the analog computing readout and sampling processing unit. These two units work collaboratively through unified control and scheduling. The core control and scheduling unit is the parallel control and neural network inference module (FPGA), which is responsible not only for non-convolutional operations and inference control in the neural network calculation but also for the unified multiplexing and scheduling of the driving circuits of the photodetector array and memristor matrix, achieving an organic combination of sensing computing and in-memory computing acceleration.

[0046] like Figure 1As shown, the signal drive and path control unit includes a multi-track power supply module, voltage output modules A and B (DAC array), path switching modules A and B (analog switch array), and an FPGA control unit. A significant feature of this unit is that it provides a reusable drive and path switching circuit for the photodetector and memristor. It outputs precise voltage signals through the multi-track power supply module and DAC array, and under the control of the path switching module, flexibly switches the drive signal to the photodetector and front-end convolutional detector array or the analog computational memristor matrix. This unit can not only apply a bias voltage to the photodetector to drive the first layer of convolution calculation, but also apply an input voltage to the memristor matrix in subsequent computation stages, achieving parallel acceleration of convolution or fully connected operations.

[0047] The analog computation readout and sampling processing unit includes a multi-track power supply module (multiplexed with the signal drive and path control unit), an analog signal filtering module (active RC array), a path switching module C, an analog signal sampling module (ADC array), a voltage output module C (DAC array), and an FPGA control unit (multiplexed with the signal drive and path control unit). This unit is primarily responsible for uniformly processing the analog output signals from the photodetector array or memristor matrix. The signals first undergo amplitude and bandwidth adjustment via a signal amplification and pre-sampling buffer module (TIA array) and an active RC filter array. Then, the path switching module C selectively feeds the signals into the ADC array for analog-to-digital conversion. Finally, the FPGA performs digital domain neural network computation and inference output. Because the signal links for the detector and memristor are shared, redundant construction of peripheral circuits is minimized, reducing system complexity and power consumption.

[0048] In the photoelectric sensing and front-end convolutional detector array section, this detector array is responsible for receiving and converting light signals of different wavelengths, covering multiple bands such as visible light and infrared. The photocurrent output of the detector array is linearly related to the light intensity and the bias voltage. Specifically, each pixel in the array responds to the intensity of the incident light and adjusts its photocurrent output according to the applied bias voltage, achieving linear adjustment of light intensity and bias voltage. This allows for direct pixel convolution calculation of light intensity × bias voltage at the hardware level. This method of performing convolution calculation directly at the sensing end can extract image features in real time, reduce the burden of back-end processing, and can adapt to different lighting environments and task requirements by adjusting the wavelength type.

[0049] In the analog computational memristor matrix section, this matrix is ​​used to implement matrix multiplication and addition operations in all convolutional layers and fully connected layers except the first layer. The conductance values ​​of the memristors are pre-written into the corresponding neural network weights. An input voltage vector is applied through a DAC array, and the output current vector generated by the memristor array is the result of the matrix operation. This result is amplified and filtered by a TIA array, then sent to an ADC array for analog-to-digital conversion, and finally passed to the FPGA for subsequent nonlinear activation and neural network inference. Because the driving circuit and detector section are multiplexed, the memristor operation can be efficiently switched with the detector's induction operation in different time slices, further improving the overall computational efficiency and resource utilization of the system.

[0050] In this embodiment, under the unified scheduling of the FPGA, the system sequentially completes stages such as weight loading, photodetector driving and acquisition, convolution window sliding to construct feature maps, memristor accelerated calculation, and result output. The entire process utilizes the circuit multiplexing characteristics of the signal driving and path control unit and the analog computing readout and sampling processing unit, enabling the photodetector and the analog computing memristor to share the same driving and readout links. This allows the first layer of convolution calculation and subsequent convolution / fully connected calculations to be completed separately without increasing hardware complexity. During the process, the system achieves seamless integration of optical signal sensing, analog multiply-accumulate operations, analog signal amplification, filtering, sampling, and digital domain neural network inference through the coordinated work of hardware modules such as the DAC array, multi-track power supply, path switching module, TIA array, active RC filter, and ADC array. The specific workflow of the system is as follows: Figure 2 As shown:

[0051] The first stage is the weight writing and initialization stage. When the system is initialized or the memristor weights need to be updated, the FPGA controls the path switching modules A and B and the voltage output modules A, B and C to apply write voltages to the rows, columns and selection terminals of the analog computation memristor matrix in sequence according to the neural network weight values ​​to be loaded. This completes the weight rewriting and calibration of the memristor array. This stage prepares the weights for the subsequent analog matrix operations on the memristors.

[0052] Next, the detector driving and single convolution window acquisition stage begins. The FPGA first switches path switching modules A and B to the photoelectric sensing and front-end convolution detector array, and outputs the gate / row polarization voltage V through voltage output module A. gThis achieves row selection activation. Simultaneously with row activation, the FPGA controls another voltage output module B to output the bias voltage amplitude corresponding to the convolution kernel to the activated column, enabling the detector pixel to physically perform a multiplication-addition (pixel-level convolution) operation of light intensity × bias voltage. The detector pixel linearly generates photocurrent based on the incident light intensity and the applied bias, and performs initial accumulation column-by-column within the array, outputting an analog current signal representing the convolution window (emphasizing the detector's linear response characteristics). Subsequently, the FPGA controls the signal amplification and pre-sampling buffer module, the active RC filter module, and the ADC array to perform current-to-voltage conversion, amplification, filtering, and analog-to-digital conversion on this analog current, completing the sampling of the analog signal for that window. The key point of this stage is that the detector performs the first layer of multiplication and accumulation simultaneously with sensing, and the amplification / filtering / sampling chain and the detector-memristor readout multiplexing are all uniformly time-controlled by the FPGA.

[0053] During the convolution window sliding and first-layer feature map construction stage, the FPGA repeatedly executes the three-step operation of row selection, column biasing, and sampling according to the convolution kernel stepping strategy to obtain the convolution results of different rows of the same convolution window. Then, it controls the voltage output module A or the corresponding DAC to slide the convolution window in the vertical or horizontal direction, repeating the aforementioned acquisition process to cover the entire input. Finally, in the digital domain, the FPGA sums the outputs of each window according to their corresponding positions (or aggregates them by adding between columns) to obtain the complete first-layer feature map. This stage illustrates how time multiplexing drives and readout resources can complete the row-by-row / column-by-column construction of the entire first-layer convolution feature map at the detector end.

[0054] After completing the first layer feature map, the system enters the memristor-accelerated computation stage for subsequent layers. When memristors are needed for analog matrix operations in subsequent convolutional or fully connected layers, the FPGA switches the drive path to the analog computation memristor array via path switching modules A and B, and applies an input voltage vector through the voltage output module, triggering the memristor array to generate an analog output of matrix multiplication and addition. The analog output of the memristor is also amplified, filtered, and sampled by the TIA, active RC, and ADC channels before being sent back to the FPGA. The FPGA performs inter-layer nonlinear processing such as pooling, normalization, and activation in the digital domain. If required by the system, other network layer calculations can be performed within the FPGA, or intermediate results can be written back to the memristor for continued analog computation. This stage highlights the reusability of the drive and readout links and the scheduling role of the FPGA among heterogeneous hardware.

[0055] Finally, in the result synthesis and output stage, after all network layer calculations and post-processing are completed, the FPGA sends the final recognition / classification results to the image processing host computer or external terminal via the host computer interface for display, recording, or further decision-making. The entire process is precisely scheduled by the FPGA on the timeline, including weight writing, detector row and column activation, convolution window sliding, memristor calling, analog readout, and digital post-processing, ensuring efficient collaboration between inductive computation (the first layer at the detector end), in-memory computation (the subsequent layers at the memristor end), and digital inference (the FPGA end).

[0056] Example 2

[0057] The system in this embodiment is divided into a photoelectric sensing and front-end convolutional detector driving architecture and an analog computing memristor matrix driving architecture. The two share some DAC arrays, analog switch arrays, signal amplification and sampling links, realizing efficient reuse and unified scheduling of hardware resources.

[0058] In such Figure 3 In the photoelectric sensing and front-end convolutional detector driving architecture shown, the detector array is arranged in a two-dimensional matrix structure, covering multiple pixel units responding to visible light and infrared bands. The array is connected to a DAC array and an analog switch array at its two ends. The FPGA controls the precise bias voltage output of these DACs and switches the analog switches to achieve row-by-row or column-by-column scanning drive of the pixels. The detector's photocurrent output is linearly related to the light intensity and the applied bias voltage, thus performing the product operation of light intensity × bias voltage at the hardware level, realizing the first layer of convolution calculation. The detector's current signal is converted into a voltage signal by the TIA array, then filtered by an active RC array, and finally enters the ADC array to complete analog-to-digital conversion, providing data input for subsequent digital processing by the FPGA.

[0059] In such Figure 4 In the analog computing-type memristor matrix driving architecture shown, the memristor matrix consists of multiple cross-array memory units, addressed and driven by word lines (WL0~WLm), bit lines (BL0~BLn), and source lines (SL0~SLm), respectively. The driving circuit also consists of a DAC array and an analog switch array (MUX), and shares hardware resources with the detector driving section. The FPGA controls the path switching to load the DAC output voltage onto the target word line or bit line, realizing the writing, updating, or inference operations of the memristor weights. In computing mode, the input voltage signal is applied to the word line, the current output of each column is amplified by the TIA array, then processed sequentially by the RC filter array and the ADC sampling array, and finally sent to the FPGA to complete the reading of the operation results of the convolutional layer or fully connected layer.

[0060] The core advantage of this architecture lies in its ability to multiplex the DAC array, analog switch array, and analog signal links at the hardware level. This satisfies both the driving and readout requirements of the first-layer convolution calculation of the photodetector and provides voltage driving and output acquisition for inference and training of the memristor matrix. This significantly reduces hardware resource consumption and improves system integration and computational efficiency. This embodiment describes the expansion and improvement of the FPGA-accelerated inductive focal plane image recognition system in future applications, demonstrating the flexibility and adaptability of the invention's architectural design.

[0061] Firstly, regarding the sensing detector array, existing systems already support multi-band detection in the visible and infrared bands. However, this system architecture is not limited to this. By replacing or integrating different types of optoelectronic materials (such as ultraviolet detectors, terahertz detectors, etc.), the spectral response range of this system can be extended to the ultraviolet, terahertz, and even X-ray bands to meet the imaging and computing needs of more special environments. The detector array size can also be expanded according to the application scenario, from the current two-dimensional matrix to a very large-scale array. Combined with row and column scanning and partitioned readout technology, higher resolution real-time sensing capabilities can be achieved.

[0062] Secondly, regarding memristor computing arrays, this system architecture can support the stacking and parallel computing of multiple memristor matrices to meet the needs of larger-scale neural network models. The expansion of the memristor array not only improves the throughput of convolutional computing, but also supports the dynamic loading and switching of various network structures through modular design, thereby adapting to the computing needs of different tasks.

[0063] In summary, this embodiment demonstrates that the system architecture of the present invention has high scalability and improvement potential. It can not only adapt to different bands and computing scales by replacing the front-end sensing devices and the back-end computing array, but also achieve rapid deployment and upgrades in different application scenarios by reusing the driver architecture and reconfigurable FPGA logic, thereby significantly extending the system's life cycle and technical competitiveness.

[0064] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. An image recognition system based on FPGA-driven sensing focal plane and memristor, characterized in that, include: The photoelectric sensing and front-end convolutional detection array acquires image data by sensing light intensity and adjusts the output photocurrent signal according to the applied bias voltage to achieve linear adjustment of light intensity and bias voltage. Thus, the pixel convolution calculation of light intensity × bias voltage, i.e. the first layer convolution operation, is directly completed at the hardware level. Analog computational memristor arrays are used to implement matrix multiplication and addition operations in convolutional layers and fully connected layers other than the first convolutional layer during neural network inference. Signal drive and path control unit is used to drive photoelectric sensing and front-end convolutional detection array or analog computing memristor array. The FPGA is used to schedule and select the corresponding pixels and memristors to participate in the operation. The analog computing readout and sampling processing unit is used to process the output of the photoelectric sensing and front-end convolutional detector array or the analog computing memristor array, and then use the FPGA to perform non-convolutional operations in the neural network inference process in the digital domain to complete the image recognition and classification task. The photoelectric sensing and front-end convolutional detector array and the analog computing memristor array share the same set of signal driving and path control unit and analog computing readout and sampling processing unit. The signal drive and path control unit includes a voltage output module, a parallel control module, and a path switching module. The voltage output module uses a DAC array and is controlled by the parallel control module to provide bias voltage to the pixels or input voltage to the memristors. The path switching module uses an analog switch array and is controlled by the parallel control module to select the corresponding pixels or memristors to participate in the operation. The parallel control module uses an FPGA to uniformly manage the working timing and data flow of the pixels and memristors through scheduling. The analog computation readout and sampling processing unit includes a signal amplification and pre-sampling buffer module, an analog signal filtering module, an analog signal sampling module, a path switching module, and a neural network inference module. The path switching module uses an analog switch array and is controlled by the neural network inference module, selecting the output of a corresponding memristor for subsequent processing. The signal amplification and pre-sampling buffer module uses a TIA array to amplify the output of the photoelectric sensing and front-end convolutional detector array or the analog computational memristor array. The analog signal filtering module uses an active RC array to filter the amplified output. The analog signal sampling module uses an ADC array to convert the filtered output into a digital signal. The neural network inference module uses an FPGA, which performs non-convolutional operations on the digital signal in the digital domain and outputs the results to complete the image recognition and classification task.

2. The image recognition system based on FPGA-driven sensing focal plane and memristor according to claim 1, characterized in that: The photoelectric sensing and front-end convolutional detection array consists of multiple pixels arranged in an array. The pixels sense the illumination intensity of incident light in different wavelength bands and adjust the output of the photocurrent signal according to the applied bias voltage. The photocurrent signal is linearly related to the illumination intensity and the bias voltage.

3. The image recognition system based on FPGA-driven sensing focal plane and memristor according to claim 1, characterized in that: In the analog computational memristor array, the conductance value of the memristor is pre-written into the corresponding neural network weights. The applied input voltage vector generates an output current vector through the memristor array, which is the result of matrix operation.

4. The image recognition system based on FPGA-driven sensing focal plane and memristor according to claim 1, characterized in that: The parallel control module and the neural network inference module share the same FPGA.

5. The image recognition system based on FPGA-driven sensing focal plane and memristor according to claim 4, characterized in that: The workflow of the image recognition system is as follows: (1) According to the neural network weight values ​​to be loaded, the FPGA controls the path switching module and the voltage output module to apply the write voltage to the rows, columns and selection terminals of the analog computing memristor matrix in sequence, so as to complete the weight rewriting and calibration of the memristor array. (2) The FPGA switches the drive to the photoelectric sensing and front-end convolution detection array through the path switching module, and uses the voltage output module to output the gate voltage to activate the pixels in the corresponding row of the array. While the row is activated, the FPGA controls another voltage output module to apply the bias voltage of the corresponding column mapping of the convolution kernel in sequence, so that the pixel physically realizes the pixel convolution operation of light intensity × bias voltage. The pixel generates photocurrent linearly according to the incident light intensity and the applied bias voltage and completes the initial accumulation in the array according to the row and column, and outputs the analog photocurrent signal representing the corresponding convolution window. Then the FPGA controls the signal amplification and sampling pre-buffer module, the analog signal filtering module and the analog signal sampling module to perform current-to-voltage conversion, amplification, filtering and analog-to-digital conversion on the analog photocurrent signal to complete the analog signal sampling of the convolution window. The FPGA slides the convolution window in the vertical or horizontal direction according to the convolution kernel stepping strategy, and repeats the above three-step operation of row selection, column bias and sampling to obtain the convolution results of different convolution windows. Finally, the FPGA sums the output of each window according to the corresponding position in the digital domain to obtain the complete feature map. (3) The FPGA switches the drive to the analog computing memristor array through the path switching module. The voltage output module applies the input voltage vector to the array, triggering the memristor array to generate an analog output of matrix multiplication and addition. The analog output of the memristor array is also sent to the FPGA after being amplified, filtered and converted by the signal amplification and sampling pre-buffering module, the analog signal filtering module and the analog signal sampling module. The FPGA completes the interlayer nonlinear processing, including pooling, normalization and activation, in the digital domain to obtain the image recognition and classification results and send them to the host computer for display and recording.

Images