A multi-hardware collaborative intelligent image processing architecture and method integrating sensing, storage, and computing

Abstract

This invention discloses a multi-hardware collaborative intelligent image processing architecture and method integrating sensing, storage, and computing, belonging to the field of edge artificial intelligence hardware and intelligent image processing technology. It includes: an MPU module, an FPGA module, and an RRAM module, tightly coupled and collaboratively connected via high-speed on-board or on-chip interconnects. The MPU module is responsible for image sensor control, system collaborative management, and nonlinear domain computation in the neural network. The FPGA module is responsible for pipeline preprocessing of image data acquired by the image sensor and control of the RRAM module. The RRAM module is responsible for analog domain multiplication and accumulation computation of the preprocessed image data in the convolutional and fully connected layers of the neural network. Through an architecture-level task mapping strategy, this invention ensures recognition accuracy while balancing processing speed and system energy consumption, providing a high-performance, high-reliability technical solution for intelligent sensing systems in resource-constrained scenarios.

Description

Technical Field

[0001] This invention belongs to the field of edge artificial intelligence hardware and intelligent image processing technology, specifically relating to a multi-hardware collaborative sensing-memory-computing integrated intelligent image processing architecture and method. Background Technology

[0002] With the rapid proliferation of unmanned security nodes, lightweight inspection robots, and small rotorcraft platforms, "edge-side local image recognition" has become a rigid requirement for the perception layer. These devices are generally subject to stringent physical constraints in terms of power consumption budget, payload, and battery life. Simultaneously, they must maintain high frame rates and low end-to-end latency in complex environments—accessible day and night, permeable to fog and dust, and tolerable to electromagnetic interference—to support flight control closed-loop or motion control logic. Furthermore, the highly diverse and rapidly iterating nature of frontline mission scenarios demands that inference hardware be able to perform model replacements, instruction set additions / removals, and sensor parameter revisions on-site, without relying on cloud computing or high-power workstations. This combination of "low power consumption × high frame rate × dynamic reconfigurability" constitutes the core technical challenge of edge image recognition systems.

[0003] Executing convolutional neural networks directly on a von Neumann CPU or MPU inevitably involves high-frequency data movement between cache and main memory: feature maps need to read and write tens of KB of weights and intermediate tensors after each convolutional layer, and memory access energy consumption is often higher than effective computational energy consumption; at the same time, instruction-level parallelism is limited, making it difficult to support the required tens of millions of MACs / s at milliwatt-level power, and frame rate and latency cannot meet the standards. Although pure FPGA solutions can use logic arrays for massively parallel MACs, high-order nonlinear algorithms such as ReLU, BatchNorm, and Soft-Max cannot be efficiently implemented within the LUT, resulting in a sharp increase in resource consumption and timing convergence pressure; once the model is upgraded, it must be resynthesized and reimplemented, resulting in long downtime and high maintenance costs. If GPUs are used instead, power consumption and heat dissipation are completely beyond the limits of small platforms, and the size and weight do not meet the requirements. Therefore, a single computing domain architecture cannot achieve a balance between power consumption, real-time performance, and flexibility.

[0004] To reduce the power consumption of convolution multiplication and accumulation, researchers have tried various two-stage heterogeneous combinations:

[0005] The FPGA+RRAM solution utilizes an in-memory computing array to handle convolution and full connectivity, but complex functions such as activation and normalization still fall on the FPGA side. Using LUTs or fixed-point polynomial approximations results in additional logic and power consumption. At the same time, the FPGA must simultaneously control a variety of devices, such as image sensors, memristor drivers—DACs, ADCs, and a large number of analog switch arrays. The control logic, image pipeline, and array timing compete for on-chip bandwidth, making it difficult to further improve the frame rate.

[0006] The CPU / MPU+RRAM solution migrates nonlinear operators to the MPU, and the convolution, multiplication, and accumulation are handled by the array. However, the feature map needs to frequently travel between the array and main memory, making the system bus a new bottleneck. The MPU also needs to manage the image sensor bias, exposure, and temperature control simultaneously, which severely squeezes the already tight I / O bandwidth.

[0007] The CPU / MPU+FPGA solution replaces RRAM with digital convolution accelerators, which reduces memory access pressure to some extent. However, the power consumption and area of ​​digital multiplication are still high. Furthermore, when the computation scale is large, the limited DSP units inside the FPGA will also reach the parallel computing bottleneck.

[0008] These dual heterogeneous paths have each made improvements, but they still cannot simultaneously meet the comprehensive indicators of low power consumption, high real-time performance, high frame rate, and flexible on-site reconstruction of edge infrared image recognition. The technical gap still exists.

[0009] In summary, existing single-domain or dual-domain computing architectures present inherent trade-offs in terms of power consumption, bandwidth, heterogeneous collaboration, and model upgrade flexibility, failing to provide edge devices with highly energy-efficient, real-time, and easily maintainable infrared image recognition capabilities. This urgently necessitates a novel multi-domain heterogeneous computing architecture capable of precise division of labor and complementary collaboration among convolutional multiplication and accumulation, complex nonlinear operations, and sensor / system scheduling, providing a technological direction for future inventions. Summary of the Invention

[0010] In view of the above, the purpose of this invention is to provide a multi-hardware collaborative intelligent image processing architecture and method integrating sensing, storage, and computing. Based on the MPU-FPGA-RRAM ternary heterogeneous computing architecture, it solves the comprehensive technical problems in edge image recognition, such as power consumption constraints, difficulty in balancing frame rate and latency, and insufficient on-site model reconfigurability. By performing complementary division of labor among the MPU, FPGA, and RRAM, it strives to achieve collaborative processing of real-time preprocessing, convolution multiplication and accumulation, nonlinear operations, and sensor management of infrared or visible light images without increasing the overall size and heat dissipation burden. This meets the engineering requirements of edge devices for high-speed real-time inference, low end-to-end latency, and rapid on-site model updates under stringent power consumption budgets.

[0011] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0012] This invention provides a multi-hardware collaborative intelligent image processing architecture integrating sensing, storage, and computing, comprising: an MPU module, an FPGA module, and an RRAM module that form a tightly coupled collaborative relationship through high-speed interconnection on board or on chip;

[0013] The MPU module is responsible for image sensor control, system collaborative management, and nonlinear domain calculations in neural networks.

[0014] The FPGA module is responsible for the pipeline preprocessing of image data acquired by the image sensor and the control of the RRAM module.

[0015] The RRAM module is responsible for the analog domain multiplication and accumulation calculation of the preprocessed image data in the convolutional and fully connected layers of the neural network.

[0016] The analog domain multiplication and accumulation calculation results of the convolutional layer and fully connected layer of the RRAM module are transmitted to the MPU module via the FPGA module. After combining the nonlinear domain calculation of the MPU module, the inference result is output to complete the image recognition processing.

[0017] Preferably, the FPGA module includes an image processing module, a memristor control module, and a data transmission module;

[0018] The image processing module is used to perform preprocessing operations on the images acquired by the image sensor, including synchronization, non-uniformity correction, median filtering, and scaling transformation.

[0019] The data transmission module is used to rearrange the preprocessed image data stream according to the format requirements of the convolutional neural network and then write it into the RRAM module.

[0020] The memristor control module is used to control the weight writing, reading, and matrix multiplication operations of the RRAM module through parallel control of the digital-to-analog converter and the analog-to-digital converter.

[0021] Preferably, the FPGA module is configured with an RJ45 interface for inputting processed image data into the RRAM module to perform analog domain multiplication and accumulation calculations, an SPI interface for high-speed instruction and data interaction with the MPU module, and a JTAG interface for firmware burning, debugging, and online updates with the MPU module.

[0022] Preferably, the RRAM module includes several RRAM arrays, adopting a 1T1R crossover structure, and each crossover unit is composed of a resistive random access memory and a gating transistor connected in series.

[0023] Preferably, the RRAM module further includes a status control and detection module, which monitors key parameters in the RRAM array, including conductance drift, write stability, and power status, and uploads the monitoring data to the MPU module in real time via an Ethernet communication link.

[0024] Preferably, in the RRAM module, the weight matrix of the neural network model is directly mapped to the conductance value of each resistive random access memory cell in the RRAM array, and the corresponding input excitation is applied as an analog voltage to the corresponding SL port, and the RRAM array performs analog domain parallel multiplication and accumulation operations of the convolutional layer or fully connected layer.

[0025] Preferably, the MPU module includes a sensor control module, a system collaborative management module, and a nonlinear domain calculation module;

[0026] The sensor control module is used to control the parameter configuration of the image sensor and to receive the output signal of the image sensor through the video parallel data interface for synchronous status monitoring and subsequent processing.

[0027] The system collaborative management module is used to dynamically update the network structure or weight allocation strategy during operation, accurately synchronize the quantized weights of the neural network to the FPGA and RRAM according to the structural mapping relationship, and coordinate the status of the FPGA and RRAM simultaneously for status monitoring and switching.

[0028] The nonlinear domain calculation module is used to perform nonlinear activation function calculations and ultimately complete the classification output or upload the inference results to the main control system.

[0029] Preferably, the image sensor output signal includes pixel clock, line synchronization, frame synchronization, and image data.

[0030] To achieve the above-mentioned objectives, this invention also provides a multi-hardware collaborative integrated sensing, storage, and computing intelligent image processing method, implemented using the aforementioned multi-hardware collaborative integrated sensing, storage, and computing intelligent image processing architecture, comprising the following steps:

[0031] Utilizing the MPU module for image sensor control, system collaborative management, and nonlinear domain computation in neural networks;

[0032] The FPGA module is used for pipeline preprocessing of image data acquired by the image sensor and control of the RRAM module.

[0033] The RRAM module is used to perform simulated domain multiplication and accumulation calculations on preprocessed image data in the convolutional and fully connected layers of a neural network.

[0034] Preferably, during image recognition, the acquired image is preprocessed by the FPGA module and then input into the neural network model. The RRAM module and MPU module work together to achieve accurate classification and recognition, and the final recognition result is returned to the host computer or execution terminal.

[0035] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0036] (1) This invention utilizes MPU to process nonlinear layers and sensor scheduling, FPGA to handle high-speed pixel pipeline and RRAM array driving, and RRAM to perform in-situ convolution multiplication and accumulation, enabling edge devices to achieve video-level frame rate and low end-to-end latency with low power consumption. Furthermore, through layered hot updates of weights, bit streams, and firmware, it takes into account the three key technical indicators of low power consumption, high real-time performance, and fast model iteration.

[0037] (2) The present invention uses medium-low density FPGA, MPU and small RRAM array to replace high power SoC / GPU, which can simplify PCB and heat dissipation design; high energy efficiency brings longer battery life; on-site upgrades and maintenance can be completed by remote firmware and weight distribution, which significantly improves on-site work efficiency.

[0038] (3) In scenarios where there is a lack of stable power grid and network coverage, such as forest fire sentry towers in nature reserves, individual soldier rescue terminals in mountainous areas, and intelligent monitoring pillars at remote intersections in cities, the architecture of this invention can be powered by small solar energy or disposable lithium batteries for a long time due to its low power consumption; millisecond-level local inference can identify smoke points, gestures for help, or abnormal traffic actions in a timely manner without cloud transmission, so as to achieve faster disaster warning, rescue call and traffic diversion response. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the overall functional architecture of the multi-hardware collaborative sensing, storage, and computing integrated intelligent image processing architecture provided in this embodiment of the invention.

[0041] Figure 2 A schematic diagram of the functional architecture of the MPU in the multi-hardware collaborative sensing, storage, and computing integrated intelligent image processing architecture provided in this embodiment of the invention;

[0042] Figure 3 This is a schematic diagram of the functional architecture of the FPGA in the multi-hardware collaborative sensing, storage, and computing integrated intelligent image processing architecture provided in this embodiment of the invention.

[0043] Figure 4 This is a schematic diagram of the functional architecture of RRAM in the multi-hardware collaborative sensing, storage, and computing integrated intelligent image processing architecture provided in this embodiment of the invention.

[0044] Figure 5This is a schematic diagram of the model mapping between RRAM and neural network in the multi-hardware collaborative sensing-memory-computing integrated intelligent image processing architecture provided in this embodiment of the invention.

[0045] Figure 6 This is a flowchart illustrating the multi-hardware collaborative sensing, storage, and computing integrated intelligent image processing method provided in this embodiment of the invention.

[0046] Figure 7 This is a neural network model diagram applied to infrared gesture recognition in the multi-hardware collaborative sensing, storage, and computing integrated intelligent image processing method provided in this embodiment of the invention.

[0047] Figure 8 This is an example diagram of the infrared gesture dataset collected in the multi-hardware collaborative sensing, storage, and computing integrated intelligent image processing method provided in this embodiment of the invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0049] The core problems to be solved by this invention include: reducing energy consumption and bandwidth conflicts caused by large-scale multiplication and accumulation operations in convolutional neural network computation; completing high-complexity nonlinear layer computation and real-time sensor control while maintaining power far lower than traditional computing architectures; avoiding frame rate decline caused by memory access wall, computation or timing resource limitations in a single computing domain; and providing a hardware layout foundation for rapid iteration of model, algorithm or sensor parameters on-site.

[0050] In view of this, embodiments of the present invention provide a multi-hardware collaborative intelligent image processing architecture and method integrating sensing, storage, and computing. The core design idea originates from an in-depth analysis of the characteristics of various computational loads in image recognition tasks, proposing to map different computational subtasks to the most suitable computing units for execution, thereby achieving optimal allocation of computing resources and a global improvement in energy efficiency. Specifically, the present invention constructs an efficient collaborative mechanism through fine decoupling and distributed collaboration of functional modules. This mechanism involves the MPU handling image sensor control, computation of complex nonlinear functions, and system scheduling; the FPGA undertaking parallel image data preprocessing and RRAM control logic; and the RRAM completing computationally intensive convolution and fully connected operations in neural networks. This structure effectively leverages the respective advantages of each computing unit in flexible control, high-throughput preprocessing, and neuromorphic simulation computing, enabling high-frame-rate, low-latency, and low-power image recognition tasks to be completed at the edge. Through an architecture-level task mapping strategy, the present invention ensures recognition accuracy while balancing processing speed and system energy consumption, providing a high-performance and high-reliability technical solution for intelligent sensing systems in resource-constrained scenarios.

[0051] like Figure 1 As shown, this embodiment provides a multi-hardware collaborative intelligent image processing architecture integrating sensing, storage, and computing. It achieves low-power image recognition computation based on the heterogeneous collaboration of an MPU (Microprocessor Unit), an FPGA (Field-Programmable Gate Array), and RRAM (Resistive Random Access Memory). These three components form a tightly coupled collaborative relationship through high-speed on-board or on-chip interconnects. Each module is linked through status control signals, image data streams, and control buses to collaboratively complete the entire process of image acquisition, preprocessing, neural network inference, and classification. The entire system adopts fully digital connectivity, and all information interaction is completed through a structured control protocol to ensure the synchronization and stability of communication between heterogeneous units. This architecture is specifically designed for real-time intelligent image recognition in power-constrained edge devices.

[0052] The main division of labor and collaborative relationships among the various parts are as follows:

[0053] 1. The MPU module is primarily responsible for system control and nonlinear domain calculations:

[0054] (1) Control the register configuration, power management and task scheduling of the camera or other image sensor;

[0055] (2) Perform nonlinear or scalar operations such as activation functions and normalization in neural networks that cannot be directly implemented by RRAM arrays;

[0056] (3) Detect and receive the operating status of the FPGA and RRAM array, and control the system's workflow;

[0057] (4) Perform fault recovery when the system malfunctions;

[0058] (5) When upgrading and switching neural network models, it is responsible for weight management, system parameter modification and FPGA bit stream upgrade.

[0059] 2. The FPGA module is mainly responsible for image pipeline processing and RRAM array control:

[0060] (1) Establish a deep pipeline for sensor image data to directly and quickly perform streaming preprocessing such as synchronization, noise reduction, and scale transformation on the original image. After on-chip buffering, the data is directly pushed to the memristor (i.e. RRAM) module to avoid off-chip storage and handling.

[0061] (2) The input characteristics are voltage applied and the result current is sampled by parallel driving of the RRAM array through DAC (digital-to-analog converter), ADC (analog-to-digital converter) and analog switches.

[0062] 3. The RRAM module is mainly responsible for simulating multiplication and accumulation calculations in the analog domain:

[0063] (1) Maintain the weights of neural network convolutional and fully connected layers in the form of electrical conductivity for a long period of time;

[0064] (2) The matrix-vector multiplication and accumulation operation is completed in the analog domain at one time, and the output is provided to the FPGA module in the form of bit line current.

[0065] In the embodiments, such as Figure 1 As shown, the image sensor module is used to sense light signals from the external environment, read electrical signals, and perform analog-to-digital conversion. Its main function is to convert received infrared or visible light image information into standard format digital image data and transmit it to a subsequent processing module (such as an FPGA) for further calculations. Simultaneously, the image sensor also has real-time feedback capabilities for current operating status information, facilitating system-level scheduling and parameter adjustments by the MPU module. This image sensor module typically integrates core sub-units such as photoelectric conversion devices, analog signal amplification circuits, analog-to-digital converters (ADCs), and timing control logic, supporting various frame rates, resolutions, and spectral response formats to meet the image acquisition needs of different application scenarios.

[0066] Furthermore, to achieve a flexible and controllable image acquisition strategy, in this embodiment, the configuration information of the image sensor (such as gain, exposure time, sampling window, triggering mode, etc.) is dynamically configured by the MPU module through standard communication interfaces (such as I2C, SPI, or MIPI), while simultaneously monitoring its operating status to achieve closed-loop control and robust scheduling. The image sensor described in this invention is not limited to a specific model or standard, and may include, but is not limited to, sensors based on CMOS (Complementary Metal-Oxide-Semiconductor), CCD (Charge-Coupled Device), and VO2max. xImage sensing components made from detection materials or structures such as vanadium oxide, InGaAs, and HgCdTe are suitable for both visible light imaging systems and short-wave, mid-wave, or long-wave infrared image acquisition systems, and have good versatility and scalability.

[0067] In the embodiments, such as Figure 2 As shown, the MPU module, as the core processor for system-level scheduling and nonlinear computation, integrates a sensor control module, a system coordination management module, and a nonlinear domain computation module. It primarily undertakes multiple tasks, including image sensor configuration, system status detection and control, neural network nonlinear layer computation, anomaly detection and recovery, and weight synchronization management. The sensor control module includes a register configuration module and a sensor status control module; the system coordination management module includes a weight management and synchronization module, an FPGA bitstream writing and status detection module, an RRAM status detection and control module, and an anomaly handling and recovery module; and the nonlinear domain computation module includes a DSP computation module and a classification result and video output module. Its functional module design is clearly structured and hierarchical, effectively supporting the stable operation and real-time response capabilities of this heterogeneous computing system.

[0068] First, the MPU establishes a configuration communication channel with the image sensor via the integrated I2C interface. Combined with the register configuration module, it can send low-level configurations such as sensor sampling parameters and operating modes, ensuring stable operation of the image acquisition unit at specified resolution, frame rate, and dynamic range. Simultaneously, the video parallel data interface receives PCLK (pixel clock), VSYNC (frame synchronization), HSYNC (line synchronization), and DATA (image data) signals from the sensor, allowing the MPU to monitor synchronization status and perform subsequent processing.

[0069] The image sensor's status is continuously monitored by the sensor status control module. Once a fault event such as frame loss or communication interruption is detected, the exception handling and recovery module is immediately invoked to perform a soft reset, parameter reconfiguration, or fault flag reporting, ensuring the system's continuous availability and anti-interference capability.

[0070] For the deployment phase of neural network models, the MPU incorporates a weight management and synchronization module. This module accurately synchronizes the quantized weights of the neural network to the FPGA and RRAM array based on the structural mapping relationship, achieving integrated weight loading for multi-layer network structures. This module communicates with the FPGA via an SPI bus interface and provides a dynamic update mechanism to adapt to task adjustments, enhancing system flexibility and scenario adaptability.

[0071] During mission execution, the MPU continuously monitors the operating status of the FPGA and RRAM, implementing status acquisition and fault reset mechanisms through JTAG and RJ45 interfaces. The status information of the FPGA and RRAM is fed back to the MPU by the FPGA bitstream writing and status detection module and the RRAM status detection and control module, respectively. Combined with the nonlinear calculation results from the DSP computing module, the classification results and video output module then complete the final classification determination and uploading. The final result can be uploaded to the host computer via serial port or USB, enabling system output visualization or further control logic closed-loop.

[0072] In summary, the MPU module, as the central control core of a heterogeneous system, plays a crucial role in system scheduling, sensor configuration, neural network control, and auxiliary computing. Its modular design ensures that the system maintains high adaptability, scalability, and stability even in power-constrained and resource-constrained edge scenarios, making it an indispensable and important component of the technical architecture of this invention.

[0073] In the embodiments, such as Figure 3 As shown, the overall structure of the FPGA module includes an image preprocessing module, a memristor control module, and a data transmission module. In the system, it primarily undertakes core tasks such as image data processing, in-memory array weight control and interaction, and result feedback. The image processing module includes image filtering, image enhancement, and image scaling modules; the memristor control module includes weight updating and writing modules; and the data transmission module includes a result reading module. Its functional modules possess highly parallel processing capabilities and flexible hardware programmability, providing crucial support for real-time image recognition tasks in heterogeneous computing architectures.

[0074] First, the FPGA receives raw image signals from the image sensor via a video parallel data interface, including PCLK, HSYNC, VSYNC, and DATA. The image data is first fed into the image filtering module, where noise suppression algorithms such as median filtering are performed to effectively eliminate thermal noise and background interference. Then, the image enhancement module is used to improve image quality, providing reliable input for subsequent neural network inference.

[0075] The enhanced image data is further compressed in real time by an image scaling module. This module uses a bilinear interpolation algorithm to scale the original image to a fixed input size to adapt to the data channel width and computational scale of the downstream RRAM array structure, ensuring a balance between overall system computational efficiency and recognition accuracy.

[0076] During the neural network deployment phase, the FPGA completes the weight programming of the RRAM array through an integrated weight update and write module. This module receives the quantized neural network weight information from the MPU, performs data format conversion based on the weight mapping strategy, and transmits it to the RRAM array via the standard Ethernet physical layer through the RJ45 interface to realize the weight programming operation. The entire process follows a row-by-row or column-by-column write strategy to ensure the accuracy of weight mapping and conductance balance.

[0077] During the recognition and inference phase, the FPGA also handles the control logic for the RRAM array. After passing through the image processing module, the image data is input to the RRAM array via an RJ45 interface to perform matrix-vector multiplication operations in the analog domain. After the RRAM operation is completed, the result reading module is responsible for reading the calculation results via Gigabit Ethernet.

[0078] In addition, the FPGA is equipped with an SPI interface for high-speed instruction and data interaction with the MPU, supporting functions such as weight synchronization and control signal delivery; it is also equipped with a JTAG interface to support firmware burning, debugging and online update functions, improving the flexibility and controllability of the system during deployment and maintenance.

[0079] In summary, the FPGA module, as the core computing unit for image preprocessing and memristor array control, is primarily based on its highly parallel architecture and programmable flexibility. FPGAs are suitable for handling real-time pixel-level operations in image streams, such as median filtering, image scaling, and format conversion. They can construct pipeline structures through parallel logic resources to achieve high throughput and low latency data processing, meeting the real-time requirements of high frame rate image processing in edge computing scenarios. Simultaneously, FPGAs possess the ability to precisely control peripheral signals and timing. By configuring different interfaces and working closely with the memristor array, they can achieve key operations such as DAC excitation control, weight writing, and calculation result acquisition. Therefore, in this system, the FPGA combines high-speed data path and hardware-level memristor control functions, making it a key component for realizing the low-power, high-efficiency heterogeneous architecture of this invention.

[0080] In the embodiments, such as Figure 4 As shown, the RRAM module is the core computing unit in the system that undertakes brain-like computing tasks. It provides high-energy-efficiency hardware acceleration support for the multiplication and accumulation calculation of convolutional layers and fully connected layers in neural networks. Its hardware structure mainly includes an RRAM cross array, a digital-to-analog / analog-to-digital converter (DAC / ADC), a multiplexer, a transimpedance amplifier, and an RJ45 communication interface. Its main functions include the state control and detection of the RRAM array, weight update and calculation result reading. The functional modules work together to achieve efficient brain-like computing.

[0081] First, the FPGA transmits the pre-processed image data signal or weight information to the RRAM module via a Gigabit Ethernet interface (RJ45). This input signal first enters the weight update and write module, which automatically determines whether to perform a weight write operation or a vector-matrix multiplication operation based on the current stage of system operation. During the model deployment stage, the module controls the analog signal generator (DAC) to apply corresponding voltage pulses at the intersection of the selected rows and columns, achieving precise control of the memristor conductance value; during the inference stage, it controls the input vector to be applied to the row terminals of the RRAM array via the DAC, simulating multiplication and accumulation operations.

[0082] The core structure of the RRAM array is built on a 1T1R (one-transistor-one-resistor) architecture, where each memristor cell consists of a selection transistor and a variable-resistance memristor connected in series. This effectively suppresses the leak path in the array and enhances the current limiting capability during programming. As a novel type of non-volatile memory device, RRAM possesses multi-state conductance control capabilities and excellent scalability. By applying specific voltage pulses, the resistance state (conduction value) can be reversibly switched between a high-resistance state (HRS) and a low-resistance state (LRS), thereby enabling analog conductance programming and analog weighted calculation functions.

[0083] During the inference process, the accumulated current at the column end is converted into a voltage signal by the transimpedance amplifier (TIA), then enters the multiplexer (MUX), and is sampled and digitized by the high-precision ADC module to form the final output voltage value, which is transmitted back to the FPGA or MPU for subsequent processing and classification.

[0084] Meanwhile, the RRAM module is equipped with an independent status control and detection module to monitor key parameters such as conductance drift, write stability, and power status in the array. This module establishes an Ethernet communication link with the MPU to upload monitoring data in real time, ensuring long-term stable operation of the system and providing self-recovery and fault tolerance capabilities.

[0085] In summary, the RRAM module plays a core role in large-scale analog matrix computation in the entire heterogeneous computing architecture. Compared with the neural network inference method implemented by traditional digital logic, it can significantly reduce power consumption, improve computational parallelism and throughput, and fully meet the dual requirements of computing power and energy efficiency in edge intelligent devices.

[0086] In the embodiments, such as Figure 5As shown, RRAM devices are constructed in a cross-array configuration to form a massively parallel computing array. This array consists of three types of control lines: word lines (WL), source lines (SL), and bit lines (BL). WL controls access enable for selected memristor cells, SL applies the analog input voltage, and BL reads the current response generated by the memristor's conduction. Each memristor cell is connected at the cross-nodes of WL, SL, and BL. The analog input voltage applied by SL and the memristor's conductance form an analog multiplication operation, and the accumulated current on all BLs reflects the result of a vector-matrix multiplication-addition (MAC) calculation.

[0087] In terms of neural network mapping, the weight matrix of the neural network model is directly mapped to the conductance values ​​of each unit in the RRAM array. Specifically, for the input weight vector of each neuron in a certain layer of the neural network, each dimension of the weight is mapped to the corresponding memristor unit in a column of the RRAM array, and the corresponding input excitation (such as image pixels or features from the previous layer) is applied as an analog voltage to the corresponding SL port. The RRAM array naturally performs multiplication and addition calculations through the analog domain, and its output current is then converted by an ADC and transmitted to the FPGA and MPU for subsequent activation function, normalization, and other processing.

[0088] In this embodiment, the RRAM array supports three basic operating modes: conductance value writing (weight writing), conductance value reading (weight verification), and multiply-accumulate operation (matrix calculation). These three operating modes are implemented through different configurations of the array WL (word line), SL (source line), BL (bit line), and peripheral circuit control logic, as detailed below:

[0089] (1) Conductivity value writing operation:

[0090] During the weighting stage, the RRAM drive control module sets the target cell's WL to a high level to select the target memristor. Simultaneously, a specific voltage pulse (forward or reverse voltage) is applied between the corresponding SL and BL to alter the charge distribution or defect states within the memristor material, thereby achieving precise setting of the conductance value. This process is based on controllable pulse modulation, employing a gradually increasing or decreasing voltage strategy to achieve multi-level adjustment of the conductance. Write accuracy is typically ensured through a bit-by-bit or cell-by-cell write-verification closed loop.

[0091] (2) Conductivity value reading operation:

[0092] During the weight verification or calibration phase, a weak read voltage (generally much smaller than the write voltage to avoid accidental writes) is applied to the target memristor. By analyzing the relationship between the read voltage applied to SL and the current response of BL, the current conductance state of the memristor is quantized using an ADC, thereby determining whether the desired weight value has been achieved. The read operation can be performed on a single cell or a column array of multiple cells to ensure accurate loading of the matrix weights.

[0093] (3) Multiply-Accumulate (MAC) operation:

[0094] During the neural network inference phase, the RRAM array performs parallel multiply-accumulate operations in the analog domain. Each component of the input vector is converted into an analog voltage and simultaneously applied to the corresponding column at the SL terminal; WL is used to select the target computation unit. The conductance value in the RRAM unit and the input voltage naturally form a multiplicative response at the physical level, the result of which is the current flowing out through BL. The current in each BL, with the cooperation of the TIA (transimpedance amplifier) ​​and ADC module, completes analog accumulation and is converted into digital output, realizing high-parallel multiply-accumulate calculations of vectors and matrices, representing convolution or fully connected layer operations in the neural network.

[0095] In summary, through the coordinated operation of the above modules, the ternary heterogeneous system provided by this invention effectively divides the image recognition task into four stages: image perception, front-end preprocessing, parallel computing, and post-processing control. Each stage is undertaken by the most suitable hardware module, achieving the goal of maximizing computing efficiency and minimizing energy consumption from the system architecture level, and significantly improving the practical performance and deployment value of edge devices for complex visual recognition tasks.

[0096] Based on the same inventive concept, such as Figure 6 As shown, this embodiment of the invention also provides a multi-hardware collaborative intelligent image processing method integrating sensing, storage, and computing, comprising the following steps:

[0097] S1 utilizes the MPU module for image sensor control, system collaborative management, and nonlinear domain computation in neural networks;

[0098] S2 utilizes the FPGA module for pipeline preprocessing of image data acquired by the image sensor and control of the RRAM module;

[0099] S3 utilizes the RRAM module to perform simulated domain multiplication and accumulation calculations on preprocessed image data in the convolutional and fully connected layers of the neural network.

[0100] During image recognition, the acquired images are preprocessed by the FPGA module and then input into the neural network model. The RRAM module and MPU module work together to achieve accurate classification and recognition, and the final recognition result is returned to the host computer or execution terminal.

[0101] In this embodiment, the infrared gesture recognition work is carried out using the multi-hardware collaborative sensing-memory-computing integrated intelligent image processing architecture and method provided by the present invention. In order to achieve efficient execution of the infrared gesture recognition task, the neural network model adopted is a lightweight convolutional neural network structure, which is adapted to the resource-constrained characteristics of low-power edge computing platforms.

[0102] Specifically, such as Figure 7 As shown, the neural network model includes two convolutional layers, two pooling layers, two fully connected layers, and multiple non-linear activation and normalization operations. The network input is a 64×64 pixel single-channel infrared image, and the output is the probability values ​​of three types of gestures (corresponding to "rising," "hovering," and "falling" states, respectively). Figure 8 The images shown represent three highly distinguishable hand gestures, namely “ascending,” “hovering,” and “landing.” These gestures exhibit stable thermal distribution characteristics and morphological differences in infrared images, which is beneficial for improving the training convergence efficiency and final recognition accuracy of neural network models.

[0103] In this embodiment, the computational tasks of each layer of the neural network are collaboratively completed by three types of computing units based on their functional characteristics: the RRAM array is responsible for performing large-scale linear multiply-accumulate calculations such as convolutional layers and fully connected layers, giving full play to its advantages of high parallelism and low power consumption; the MPU mainly undertakes nonlinear and logic control operations such as normalization, activation functions, flattening, and softmax, and participates in light-load computational tasks; the FPGA serves as the system scheduling and control core, responsible for image data preprocessing, driving control of the RRAM array, and data scheduling and management between modules, thereby constructing a heterogeneous neural network computing system with reasonable division of labor and high collaborative efficiency. By delegating the weight calculations of convolutional and fully connected layers to the RRAM, handling nonlinear and control operations to the MPU, and coordinating image preprocessing and control operations by the FPGA, high-energy-efficiency operation of the neural network on resource-constrained edge devices is effectively achieved, especially suitable for scenarios such as real-time infrared image recognition, and has significant practical and promotional value.

[0104] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-hardware collaborative, sensor-computer integrated, intelligent image processing architecture, characterized in that, The application relates to an image recognition system based on a tightly coupled co-processor architecture. The MPU module is responsible for image sensor control, system co-management and nonlinear domain calculation in a neural network, and comprises a sensor control module, a system co-management module and a nonlinear domain calculation module; the sensor control module is used for controlling image sensor parameter configuration and receiving image sensor output signals through a video parallel data interface to perform synchronous state monitoring and subsequent processing; the system co-management module is used for dynamically updating a network structure or a weight distribution strategy during operation, accurately synchronizing neural network quantization weights to the FPGA and the RRAM according to a structure mapping relationship, and synchronously coordinating FPGA and RRAM states to perform state monitoring and switching; and the nonlinear domain calculation module is used for completing nonlinear activation function operation and finally completing classification output or uploading inference results to a host system. The FPGA module is responsible for image data pipeline preprocessing of an image sensor and control of the RRAM module, and comprises an image processing module, a memristor control module and a data transmission module; the image processing module is used for performing preprocessing operations including synchronization, non-uniformity correction, median filtering and scale transformation on images acquired by the image sensor; the data transmission module is used for rearranging image data streams after preprocessing according to requirements of a convolutional neural network format and writing the image data streams into the RRAM module; and the memristor control module is used for realizing weight writing, reading and matrix multiplication operation control of the RRAM module through parallel control of digital-analog converters and analog-digital converters. The RRAM module is responsible for analog domain multiplication and accumulation calculation of a convolutional layer and a fully connected layer in a neural network of preprocessed image data, and comprises a state control and detection module used for monitoring key parameters including conductance drift, writing stability and power state in the RRAM array and uploading monitoring data to the MPU module in real time through an Ethernet communication link. The analog domain multiplication and accumulation calculation results of the convolutional layer and the fully connected layer of the RRAM module are transmitted to the MPU module through the FPGA module, inference results are output after combination of nonlinear domain calculation of the MPU module, and image recognition processing is completed. In the FPGA module, an RJ45 interface is configured to input processed image data into the RRAM module to perform analog domain multiplication and accumulation calculation, an SPI interface is configured to perform high-speed instruction and data interaction with the MPU module, and a JTAG interface is configured to perform firmware programming, debugging and online updating with the MPU module.

2. The multi-hardware co-design integrated image processing architecture of claim 1, wherein, The RRAM module comprises a plurality of RRAM arrays, adopts a 1T1R cross structure, and each cross point unit is composed of a resistive random access memory and a gating transistor in series.

3. The multi-hardware co-design integrated image processing architecture of claim 1, wherein, In the RRAM module, a weight matrix of a neural network model is directly mapped to conductance values of each resistive random access memory unit in the RRAM array, corresponding input excitation is applied to a corresponding SL port as an analog voltage, and analog domain parallel multiplication and accumulation operation of a convolutional layer or a fully connected layer is performed by the RRAM array.

4. The multi-hardware co-design integrated image processing architecture of claim 1, wherein, ​ 5. The multi-hardware co-design integrated skinput intelligent image processing architecture according to claim 1, wherein, The image sensor output signal includes a pixel clock, a line synchronization, a frame synchronization and image data.

6. A multi-hardware collaborative neuromorphic computing integrated intelligent image processing method, implemented by using the multi-hardware collaborative neuromorphic computing integrated intelligent image processing architecture of any one of claims 1-5. The method comprises the following steps: The MPU module is used for image sensor control, system coordination management and nonlinear domain calculation in the neural network; The FPGA module is used for pipeline preprocessing of the image data collected by the image sensor and control of the RRAM module; The RRAM module is used for analog domain multiplication and accumulation calculation of the preprocessed image data in the convolution layer and the full connection layer of the neural network.

7. The multi-hardware co-design and neuromorphic computing integrated intelligent image processing method according to claim 6, characterized in that, In the image recognition, the collected image is input into the neural network model after preprocessing by the FPGA module, and accurate classification and recognition are realized through the cooperative calculation of the RRAM module and the MPU module, and the final recognition result is returned to the host computer or the execution end.

Images