Multi-hardware collaborative sensing, storage and calculation integrated intelligent image processing architecture and method

Abstract

The invention discloses a multi-hardware collaborative sensing, storage and calculation integrated intelligent image processing architecture and method, and belongs to the technical field of edge artificial intelligence hardware and intelligent image processing, and the architecture comprises an MPU module, an FPGA module and an RRAM module which form a tight coupling collaborative relationship through plate pole or on-chip high-speed interconnection; the MPU module is responsible for image sensor control, system collaborative management and nonlinear domain calculation in a neural network; the FPGA module is responsible for pipeline preprocessing of image data acquired by the image sensor and control of the RRAM module; and the RRAM module is responsible for analog domain multiply-accumulate calculation of the preprocessed image data in a convolutional layer and a full connection layer in the neural network. According to the invention, through an architecture-level task mapping strategy, while the recognition precision is ensured, the processing speed and the system energy consumption are considered, and a high-performance and high-reliability technical solution path is provided for an intelligent sensing system in a resource-limited scene.

Description

Technical Field

[0001] The present invention belongs to the field of edge artificial intelligence hardware and intelligent image processing technology, and specifically relates to an intelligent image processing architecture and method that integrates sensing, storage, and computing with multi-hardware collaboration. Background Art

[0002] With the rapid adoption of unmanned security nodes, lightweight inspection robots, and small rotorcraft platforms, "local image recognition on the edge" has become a rigid requirement for the perception layer. These devices are generally subject to stringent physical constraints such as power budget, payload, and endurance. At the same time, they must maintain high frame rates and low end-to-end latency in complex environments that are usable day and night, can penetrate fog and dust, and can withstand electromagnetic interference, in order to support closed-loop flight control or motion control logic. Furthermore, frontline mission scenarios are highly diverse and rapidly iterating, requiring inference hardware to be able to complete model replacements, instruction set additions and subtractions, and sensor parameter revisions on-site without relying on cloud computing or high-power workstations. This combination of "low power consumption × high frame rate × dynamic reconfigurability" constitutes the core technical pain point of edge image recognition systems.

[0003] Directly executing convolutional neural networks on a von Neumann CPU or MPU inevitably involves frequent data transfers between cache and main memory. After each convolution layer, feature maps require reading and writing tens of KB of weights and intermediate tensors, resulting in memory access energy consumption often exceeding the effective computational energy consumption. Furthermore, limited instruction-level parallelism (ILP) makes it difficult to support the required tens of millions of MACs / s within milliwatt power consumption, resulting in sub-optimal frame rates and latency. While pure FPGA solutions can utilize logic arrays for massively parallel MAC operations, high-order nonlinear algorithms such as ReLU, BatchNorm, and Soft-Max cannot be efficiently implemented within the LUTs, significantly increasing resource utilization and timing closure pressure. Model upgrades require re-synthesis and re-implementation, resulting in significant downtime and high maintenance costs. Using a GPU instead would significantly exceed the power consumption and heat dissipation requirements of a small platform, and the size and weight would be unacceptable. Therefore, a single compute domain architecture cannot achieve a balance between power consumption, real-time performance, and flexibility.

[0004] To reduce the power consumption of convolution multiplication and accumulation, researchers have tried a variety of two-level heterogeneous combinations: The FPGA+RRAM solution uses a storage-computing array to handle convolution and full connectivity. However, complex functions such as activation and normalization still fall on the FPGA side, and using LUTs or fixed-point polynomial approximations results in additional logic and power consumption. Simultaneously, the FPGA must control a large number of devices, including image sensors, memristor drivers—DACs and ADCs—and a large number of analog switch arrays. Control logic, image pipelines, and array timing compete for on-chip bandwidth, making it difficult to further increase the frame rate.

[0005] The CPU / MPU+RRAM solution migrates nonlinear operators to the MPU, and convolution, multiplication, and accumulation are performed by the array. However, feature maps need to frequently travel back and forth between the array and main memory, and the system bus becomes a new bottleneck. The MPU also has to simultaneously manage image sensor bias, exposure, and temperature control, severely squeezing the already tight I / O bandwidth.

[0006] The CPU / MPU+FPGA solution replaces RRAM with a digital convolution accelerator. While this alleviates memory access pressure to some extent, the power consumption and area of ​​digital multiplication remain high. Furthermore, when the computation scale is large, the limited DSP units within the FPGA will reach a parallel computing bottleneck. These dual heterogeneous paths have their own improvements, but they still cannot simultaneously meet the comprehensive indicators of edge infrared image recognition such as low power consumption, high real-time performance, high frame rate, and flexible on-site reconstruction, and the technical gap still exists.

[0007] In summary, existing single-domain or dual-domain computing architectures face a trade-off between power consumption, bandwidth, heterogeneous collaboration, and model upgrade flexibility, failing to provide edge devices with both energy-efficient, real-time, and maintainable infrared image recognition capabilities. This urgently requires a new multi-domain heterogeneous computing architecture that precisely divides and complements convolutional multiplication and accumulation, complex nonlinear operations, and sensor / system scheduling, providing a technical direction for subsequent inventions. Summary of the Invention

[0008] In view of the above, the purpose of the present invention is to provide a multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture and method, which solves the comprehensive technical problems such as power consumption limitation, difficulty in balancing frame rate and latency, and insufficient on-site reconfiguration capability of the model in the edge image recognition process based on the MPU-FPGA-RRAM ternary heterogeneous computing architecture. By performing functional complementary division of labor among MPU, FPGA and RRAM, we strive to achieve collaborative processing of real-time preprocessing, convolution multiplication and accumulation, nonlinear operations and sensor management of infrared or visible light images without increasing the volume of the whole machine and the heat dissipation burden, thereby meeting the engineering requirements of edge devices for high-speed real-time reasoning, low end-to-end latency and rapid update of on-site models under strict power consumption budgets.

[0009] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: An embodiment of the present invention provides a multi-hardware collaborative sensing, storage, and computing integrated intelligent image processing architecture, including: an MPU module, an FPGA module, and an RRAM module that form a tightly coupled collaborative relationship through on-board or on-chip high-speed interconnection; The MPU module is responsible for image sensor control, system coordination management and nonlinear domain calculation in neural networks; The FPGA module is responsible for pipeline preprocessing of image data collected by the image sensor and control of the RRAM module; The RRAM module is responsible for the analog domain multiplication and accumulation calculation of the pre-processed image data in the convolutional layer and the fully connected layer in the neural network; The analog domain multiplication and accumulation calculation results of the convolutional layer and fully connected layer of the RRAM module are transmitted to the MPU module through the FPGA module. After combining with the nonlinear domain calculation of the MPU module, the inference results are output to complete the image recognition processing.

[0010] Preferably, the FPGA module includes an image processing module, a memristor control module and a data transmission module; The image processing module is used to perform preprocessing operations including synchronization, non-uniformity correction, median filtering and scale transformation on the image acquired by the image sensor; The data transmission module is used to rearrange the pre-processed image data stream according to the format requirements of the convolutional neural network and then write it into the RRAM module; The memristor control module is used to realize weight writing, reading and matrix multiplication operation control of the RRAM module by parallel controlling the digital-to-analog converter and the analog-to-digital converter.

[0011] Preferably, in the FPGA module, an RJ45 interface is configured for inputting the processed image data into the RRAM module to perform analog domain multiplication and accumulation calculations, an SPI interface is configured for high-speed instruction and data interaction with the MPU module, and a JTAG interface is configured for firmware burning, debugging and online updating with the MPU module.

[0012] Preferably, the RRAM module includes several RRAM arrays, adopting a 1T1R cross structure, and each intersection unit is composed of a resistive random access memory and a gate transistor connected in series.

[0013] Preferably, the RRAM module further includes a state control and detection module for monitoring key parameters of the RRAM array including conductivity drift, write stability, and power supply status, and uploading the monitoring data to the MPU module in real time via an Ethernet communication link.

[0014] Preferably, in the RRAM module, the weight matrix of the neural network model is directly mapped to the conductance value of each resistive random access memory unit in the RRAM array, and the corresponding input stimulus is applied to the corresponding SL port as an analog voltage, and the RRAM array performs analog domain parallel multiplication and accumulation operations of the convolution layer or the fully connected layer.

[0015] Preferably, the MPU module includes a sensor control module, a system collaborative management module and a nonlinear domain calculation module; The sensor control module is used to control the image sensor parameter configuration and receive the image sensor output signal through the video parallel data interface to perform synchronous status monitoring and subsequent processing; The system collaborative management module is used to dynamically update the network structure or weight distribution strategy during operation, accurately synchronize the quantized weights of the neural network to the FPGA and RRAM based on the structural mapping relationship, and synchronously coordinate the FPGA and RRAM states for state monitoring and switching; The nonlinear domain calculation module is used to complete the nonlinear activation function operation and finally complete the classification output or upload the inference result to the main control system.

[0016] Preferably, the image sensor output signal includes pixel clock, line synchronization, frame synchronization and image data.

[0017] To achieve the above-mentioned purpose, an embodiment of the present invention further provides a multi-hardware collaborative sensing, storage and computing integrated intelligent image processing method, which is implemented using the above-mentioned multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture, including the following steps: Use MPU modules for image sensor control, system collaborative management, and nonlinear domain calculations in neural networks; Use FPGA modules to perform pipeline preprocessing of image data collected by image sensors and control of RRAM modules; The RRAM module is used to perform analog domain multiplication and accumulation calculations on the preprocessed image data in the convolutional layer and fully connected layer of the neural network.

[0018] Preferably, when performing image recognition, the collected image is pre-processed by the FPGA module and then input into the neural network model, and accurate classification and recognition are achieved through collaborative calculation by the RRAM module and the MPU module, and the final recognition result is returned to the host computer or execution end.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention utilizes MPU to process nonlinear layers and sensor scheduling, FPGA to be responsible for high-speed pixel pipeline and RRAM array driving, and RRAM to perform in-situ convolution and multiplication, so that edge devices can achieve video-level frame rate and low end-to-end latency at low power consumption. Through layered hot updates of weights, bit streams and firmware, the three key technical indicators of low power consumption, high real-time performance and rapid model iteration are taken into account.

[0020] (2) The present invention uses medium-low density FPGA, MPU and small RRAM array to replace high-power SoC / GPU, which can simplify the design of PCB and heat sink; high energy efficiency brings longer battery life; upgrades and maintenance can be completed on-site through remote firmware and weight distribution, significantly improving on-site work efficiency.

[0021] (3) In scenarios where there is a lack of stable power grid and network coverage, such as forest fire sentry towers in nature reserves, individual rescue terminals in mountainous areas, and smart monitoring poles at remote intersections in cities, the architecture of the present invention can be powered by small solar energy or disposable lithium batteries for a long time due to its low power consumption; millisecond-level local inference can promptly identify smoke points, hand gestures for help, or abnormal traffic movements without the need for cloud backhaul, thus achieving faster disaster warnings, rescue calls, and traffic diversion responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a schematic diagram of the overall functional architecture of a multi-hardware collaborative sensing, storage, and computing integrated intelligent image processing architecture provided by an embodiment of the present invention; Figure 2 Schematic diagram of the functional architecture of the MPU in the multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of the functional architecture of the FPGA in the multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture provided by an embodiment of the present invention; Figure 4 Schematic diagram of the functional architecture of RRAM in the multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture provided by an embodiment of the present invention; Figure 5 This is a schematic diagram of the model mapping between RRAM and neural network in the multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture provided by an embodiment of the present invention; Figure 6 1 is a flow chart of a multi-hardware collaborative sensing, storage and computing integrated intelligent image processing method provided by an embodiment of the present invention; Figure 7 This is a neural network model diagram for infrared gesture recognition in the multi-hardware collaborative sensing, storage and computing integrated intelligent image processing method provided by an embodiment of the present invention; Figure 8 This is an example diagram of an infrared gesture data set collected in the multi-hardware collaborative sensing, storage and computing integrated intelligent image processing method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0025] The core issues to be addressed by the present invention include: reducing energy consumption and bandwidth conflicts caused by large-scale multiplication and accumulation operations in convolutional neural network calculations; completing high-complexity nonlinear layer calculations and real-time sensor control while maintaining power far lower than traditional computing architectures; avoiding frame rate drops in a single computing domain due to memory access walls, computing or timing resource limitations; and providing a hardware layout foundation that can complete rapid iteration of models, algorithms or sensor parameters at the mission site.

[0026] In view of this, the embodiment of the present invention provides a multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture and method. The core design idea is derived from the in-depth analysis of the characteristics of multiple types of computing loads in image recognition tasks, and it is proposed to map different computing subtasks to the most suitable computing units for execution, thereby achieving the optimal configuration of computing resources and the overall improvement of energy efficiency. Specifically, the present invention constructs an efficient collaboration mechanism through the fine decoupling and distributed collaboration of functional modules, in which the MPU is responsible for image sensor control, the calculation and system scheduling of complex nonlinear functions, the FPGA is responsible for parallel preprocessing of image data and RRAM control logic, and the RRAM completes computationally intensive convolution and fully connected operations in neural networks. This structure effectively brings into play the respective advantages of each computing unit in flexible control, high-throughput preprocessing and brain-like simulation computing, and realizes the completion of high frame rate, low latency and low power consumption image recognition tasks at the edge. Through the architecture-level task mapping strategy, the present invention takes into account both processing speed and system energy consumption while ensuring recognition accuracy, and provides a high-performance and high-reliability technical solution path for intelligent perception systems in resource-constrained scenarios.

[0027] like Figure 1 As shown, the embodiment provides a multi-hardware collaborative sensing, storage, and computing integrated intelligent image processing architecture. This architecture implements low-power image recognition computing based on the heterogeneous collaboration of an MPU (microprocessor unit), an FPGA (field programmable gate array), and RRAM (resistive random access memory). These three components form a tightly coupled collaborative relationship through high-speed interconnects on-board or on-chip. The modules are linked via state control signals, image data streams, and control buses to collaboratively complete the entire process of image acquisition, preprocessing, neural network inference, and classification. The entire system utilizes fully digital connectivity, and all information exchange is accomplished through a structured control protocol to ensure synchronization and stability between heterogeneous units. This architecture is specifically designed for real-time intelligent image recognition within power-constrained edge devices.

[0028] The main division of labor and coordination relationship among various parts are as follows: 1. The MPU module is mainly responsible for system control and nonlinear domain calculation: (1) Control register configuration, power management, and task scheduling of cameras or other image sensors; (2) Performing nonlinear or scalar operations such as activation functions and normalization in neural networks that cannot be directly implemented by RRAM arrays; (3) Detect and receive the operating status of the FPGA and RRAM array, and control the system workflow; (4) When the system works abnormally, perform abnormal recovery; (5) When the neural network model is upgraded and switched, it is responsible for weight management, system parameter modification and FPGA bit stream upgrade.

[0029] 2. The FPGA module is mainly responsible for image pipeline processing and RRAM array control: (1) Establish a deep pipeline processing line for sensor image data, directly and quickly complete the streaming preprocessing of the original image, such as synchronization, denoising, and scaling, and directly push it to the memristor (i.e., RRAM) module after on-chip buffering, avoiding off-chip storage and transportation; (2) The RRAM array is driven in parallel through DAC (digital-to-analog converter), ADC (analog-to-digital converter) and analog switches to achieve voltage loading of input characteristics and result current sampling.

[0030] 3. The RRAM module is mainly responsible for analog domain multiplication and accumulation calculations: (1) Maintain the weights of the convolutional and fully connected layers of the neural network in the form of conduction for a long time; (2) Complete the matrix-vector multiplication and accumulation operation in the analog domain at one time, and provide the output to the FPGA module in the form of bit line current.

[0031] In the embodiment, Figure 1 As shown, the image sensor module is used to realize the perception of light signals in the external environment, the reading of electrical signals and analog-to-digital conversion. Its main function is to convert the received infrared or visible light image information into digital image data in a standard format and transmit it to the post-processing module (such as FPGA) for subsequent calculations. At the same time, the image sensor also has the ability to provide real-time feedback on the current operating status information, which facilitates the MPU module to perform system-level operation scheduling and parameter adjustment. The image sensor module usually integrates core sub-units such as photoelectric conversion devices, analog signal amplification circuits, analog-to-digital conversion circuits (ADCs), and timing control logic. It can support multiple frame rates, resolutions, and spectral response forms to meet the image acquisition requirements in different application scenarios.

[0032] Furthermore, in order to achieve a flexible and controllable image acquisition strategy, the configuration information of the image sensor in this embodiment (such as gain, exposure time, sampling window, trigger mode, etc.) is dynamically configured by the MPU module through a standard communication interface (such as I2C, SPI or MIPI), while monitoring its operating status to achieve closed-loop control and robust scheduling. The image sensor of the present invention is not limited to a specific model or format, and can be selected from, but not limited to, those based on CMOS (complementary metal oxide semiconductor), CCD (charge coupled device), VO x Image sensing components implemented with detection materials or structures such as InGaAs (vanadium oxide), HgCdTe (mercury cadmium telluride), etc. are suitable for visible light imaging systems as well as short-wave, medium-wave or long-wave infrared image acquisition systems, and have good versatility and scalability.

[0033] In the embodiment, Figure 2 As shown, the MPU module serves as the core processor for system-level scheduling and nonlinear computation. It integrates a sensor control module, a system collaborative management module, and a nonlinear domain computation module. It primarily handles multiple tasks, including image sensor configuration, system state detection and control, neural network nonlinear layer computation, anomaly detection and recovery, and weight synchronization management. The sensor control module includes a register configuration module and a sensor state control module; the system collaborative management module includes a weight management and synchronization module, an FPGA bitstream writing and state detection module, an RRAM state detection and control module, and an anomaly handling and recovery module; and the nonlinear domain computation module includes a DSP operation module and a classification result and video output module. Its functional module design has a clear structure and hierarchy, effectively supporting the stable operation and real-time responsiveness of this heterogeneous computing system.

[0034] First, the MPU establishes a configuration communication channel with the image sensor via the integrated I2C interface. Combined with the register configuration module, it can implement low-level configurations such as sensor sampling parameters and operating modes, ensuring stable operation of the image acquisition unit at the specified resolution, frame rate, and dynamic range. Simultaneously, the video parallel data interface receives the PCLK (pixel clock), VSYNC (frame synchronization), HSYNC (line synchronization), and DATA (image data) signals output by the sensor, enabling the MPU to monitor synchronization status and perform subsequent processing.

[0035] The status of the image sensor is continuously monitored by the sensor status control module. Once a fault event such as frame loss or communication interruption is detected, the exception handling and recovery module is immediately called to perform a soft reset, parameter reconfiguration, or fault flag reporting to ensure the system's continuous availability and anti-interference capabilities.

[0036] During the neural network model deployment phase, the MPU's built-in weight management and synchronization module precisely synchronizes the neural network's quantized weights to the FPGA and RRAM array based on structural mapping relationships, enabling integrated weight loading for multi-layer network structures. This module communicates with the FPGA via an SPI bus interface and provides a dynamic update mechanism to adapt to task adjustments, enhancing system flexibility and scenario adaptability.

[0037] During task execution, the MPU continuously monitors the operational status of the FPGA and RRAM, implementing status acquisition and fault reset mechanisms through JTAG and RJ45 interfaces. FPGA and RRAM status information is fed back to the MPU by the FPGA bitstream write and status detection module and the RRAM status detection and control module, respectively. Combined with the nonlinear calculation results of the DSP operation module, the classification results and video output module complete the final recognition category determination and upload. The final results can be uploaded to the host computer via the serial port or USB, enabling system output visualization or further control logic closure.

[0038] In summary, the MPU module, as the central control core in heterogeneous systems, plays a key role in system scheduling, sensor configuration, neural network control, and auxiliary computing. Its modular design ensures the system's high adaptability, scalability, and stability in edge scenarios with limited power consumption and resources, making it an indispensable component of the technical architecture of this invention.

[0039] In the embodiment, Figure 3 As shown in Figure 1, the overall structure of the FPGA module includes an image preprocessing module, a memristor control module, and a data transmission module. Within the system, these modules primarily handle core tasks such as image data processing, memory array weight control and interaction, and computation result transmission. The image processing module includes image filtering, image enhancement, and image scaling modules; the memristor control module includes a weight update and write module; and the data transmission module includes a computation result reading module. These functional modules possess highly parallel processing capabilities and flexible hardware programmability, providing critical support for real-time image recognition tasks within heterogeneous computing architectures.

[0040] First, the FPGA receives the raw image signal from the image sensor through the video parallel data interface, including PCLK, HSYNC, VSYNC, and DATA. The image data is first fed into the image filtering module, which performs noise suppression algorithms such as median filtering to effectively eliminate thermal noise and background interference. The image enhancement module then improves image quality, providing reliable input for subsequent neural network inference.

[0041] The enhanced image data is further compressed in real time by the image scaling module. This module uses a bilinear interpolation algorithm to scale the original image to a fixed input size to accommodate the data channel width and computational scale of the downstream RRAM array structure, ensuring a balance between overall system computational efficiency and recognition accuracy.

[0042] During the neural network deployment phase, the FPGA completes weight programming of the RRAM array via an integrated weight update and write module. This module receives quantized neural network weight information from the MPU, converts the data format based on the weight mapping strategy, and transmits it to the RRAM array via an RJ45 interface using a standard Ethernet physical layer to implement weight programming. The entire process follows a row-by-row or column-by-column write strategy to ensure weight mapping accuracy and conductance balance.

[0043] During the recognition and inference phase, the FPGA also handles the control logic for the RRAM array. After passing through the image processing module, image data is input to the RRAM array via an RJ45 interface to perform analog-domain matrix-vector multiplication. After the RRAM operation is complete, the result reading module is responsible for reading the result out via Gigabit Ethernet.

[0044] In addition, the FPGA is equipped with an SPI interface for high-speed instruction and data interaction with the MPU, supporting functions such as weight synchronization and control signal distribution; at the same time, it is equipped with a JTAG interface to support firmware burning, debugging and online update functions, thereby improving the flexibility and controllability of the system during deployment and maintenance stages.

[0045] In summary, the FPGA module serves as the core computing unit for image preprocessing and memristor array control, mainly based on its highly parallel architectural characteristics and programmable flexibility. FPGA is suitable for processing real-time pixel-level operations in image streams, such as median filtering, image scaling, and format conversion. It can build a pipeline structure through parallel logic resources to achieve high-throughput, low-latency data processing, meeting the real-time requirements for high-frame-rate image processing in edge scenarios. At the same time, FPGA has the ability to accurately control peripheral signals and timing, and can closely coordinate with the memristor array by configuring different interfaces to achieve key operations such as DAC excitation control, weight writing, and calculation result acquisition. Therefore, FPGA has both high-speed data paths and hardware-level memristor control functions in this system, and is a key component for realizing the low-power, high-efficiency heterogeneous architecture of the present invention.

[0046] In the embodiment, Figure 4As shown in the figure, the RRAM module is the core computing unit in the system that undertakes brain-like computing tasks. It provides energy-efficient hardware acceleration support for the multiplication and accumulation calculations of the convolutional layer and the fully connected layer in the neural network. Its hardware structure mainly includes an RRAM crossbar array, a digital-to-analog / analog-to-digital converter (DAC / ADC), a multi-channel analog switch, a transimpedance amplifier, and an RJ45 communication interface. The main functions it implements include state control and detection of the RRAM array, weight update, and reading of calculation results. The various functional modules work together to achieve efficient brain-like computing.

[0047] First, the FPGA transmits the pre-processed image data signal or weight information to the RRAM module via a Gigabit Ethernet interface (RJ45). This input signal first enters the weight update and write module, which automatically determines whether to perform a weight write operation or a vector-matrix multiplication operation based on the current system operation stage. During the model deployment phase, the module controls the analog signal generator (DAC) to apply corresponding voltage pulses at the intersection of the selected rows and columns to achieve precise control of the memristor conductance. During the inference phase, the control input vector is applied to the RRAM array row end through the DAC, simulating a multiplication-accumulation operation.

[0048] The core structure of an RRAM array is based on a 1T1R (one-transistor-one-resistor) architecture. Each memristor cell consists of a select transistor and a variable-resistance memristor in series, effectively suppressing leakage paths in the array and enhancing current limiting during programming. As a new type of non-volatile memory device, RRAM boasts multi-state conductance control and excellent scalability. By applying specific voltage pulses, the resistance state (on-state value) can be reversibly switched between a high-resistance state (HRS) and a low-resistance state (LRS), enabling analog conductance programming and analog weighted computing.

[0049] During the inference process, the accumulated current at the column end is converted into a voltage signal through a transimpedance amplifier (TIA), enters a multiplexer (MUX), and is sampled and digitized by a high-precision ADC module to form the final output voltage value, which is then transmitted back to the FPGA or MPU for subsequent processing and classification.

[0050] The RRAM module also includes an independent state control and detection module to monitor key parameters within the array, such as conductivity drift, write stability, and power status. This module establishes an Ethernet communication link with the MPU and uploads monitoring data in real time, ensuring long-term stable system operation, self-recovery, and fault tolerance.

[0051] In summary, RRAM modules play a core role in large-scale analog matrix computing within the entire heterogeneous computing architecture. Compared to traditional neural network inference methods implemented using digital logic, they can significantly reduce power consumption, increase computational parallelism and throughput, and fully leverage the dual demands of computing power and energy efficiency in edge smart devices.

[0052] In the embodiment, Figure 5 As shown in Figure 1, RRAM devices are constructed as a large-scale parallel computing array in a crossbar array. The array consists of three types of control lines: word lines (WL), source lines (SL), and bit lines (BL). The WL controls the access enable of the selected memristor cell, the SL is used to apply analog input voltage, and the BL is used to read the current response generated by the memristor conduction. Each memristor cell is connected to the intersection of the WL, SL, and BL. The analog input voltage applied by the SL and the conductance of the memristor form an analog multiplication operation. The accumulated current on all BLs reflects the result of the vector-matrix multiplication and addition (MAC) calculation.

[0053] In terms of neural network mapping, the neural network model's weight matrix is ​​directly mapped to the conductance values ​​of each cell in the RRAM array. Specifically, for each neuron's input weight vector in a given layer of the neural network, each weight dimension is mapped to a corresponding memristor cell in a column of the RRAM array. The corresponding input stimulus (such as an image pixel or a feature from the previous layer) is applied as an analog voltage to the corresponding SL port. The RRAM array naturally performs multiplication and addition calculations in the analog domain, and its output current is then converted by the ADC and transmitted to the FPGA and MPU for subsequent processing such as activation functions and normalization.

[0054] In the embodiment, the RRAM array supports three basic operating modes: conductance value writing (weight writing), conductance value reading (weight verification), and multiply-accumulate operations (matrix calculations). These three operating modes are achieved by different configurations of the array's WL (word lines), SL (source lines), BL (bit lines), and peripheral circuit control logic, as described below: (1) Conductance value writing operation: During the weight write phase, the RRAM driver control module sets the WL of the target cell to a high level to enable the target memristor. Simultaneously, a specific voltage pulse (forward or reverse voltage) is applied between the corresponding SL and BL to alter the charge distribution or defect states within the memristor material, thereby precisely setting the conductance. This process is based on controllable pulse modulation, employing a gradually increasing or decreasing voltage strategy to achieve multi-level conductance adjustment. Write accuracy is typically ensured through a bit-by-bit or cell-by-cell write-verify closed loop.

[0055] (2) Conductivity value reading operation: During the weight verification or calibration phase, a weak read voltage (typically much lower than the write voltage to avoid miswriting) is applied to the target memristor. By comparing the read voltage applied to the SL with the current response of the BL, the ADC quantifies the current conductance state of the memristor, thereby determining whether the desired weight value has been achieved. The read operation can be applied to a single cell or a column-oriented array of multiple cells, ensuring that the matrix weights are accurately loaded.

[0056] (3) Multiply-accumulate operation (MAC): During the neural network inference phase, the RRAM array performs analog-domain parallel multiplication and accumulation operations. Each component of the input vector is converted to an analog voltage and synchronously applied to the corresponding column on the SL. The WL is used to select the target computational unit. The conductance value in the RRAM cell and the input voltage naturally form a multiplication response at the physical level, resulting in a current flowing through the BL. The current of each BL is analog-accumulated and converted to a digital output through the cooperation of the TIA (transimpedance amplifier) ​​and ADC module, achieving highly parallel vector and matrix multiplication and accumulation, representing the convolution or fully connected layer operations in the neural network.

[0057] To sum up, through the coordinated cooperation of the above modules, the ternary heterogeneous system provided by the present invention effectively divides the image recognition task into four stages in structure: image perception, front-end preprocessing, parallel computing and post-processing control. Each stage is undertaken by the most suitable hardware module, achieving the goal of maximizing computing efficiency and minimizing energy consumption from the system architecture level, and significantly improving the practical performance and deployment value of edge devices for complex visual recognition tasks.

[0058] Based on the same inventive concept, Figure 6 As shown, an embodiment of the present invention further provides a multi-hardware collaborative sensing, storage and computing integrated intelligent image processing method, comprising the following steps: S1, uses the MPU module to control the image sensor, coordinate the system management, and perform nonlinear domain calculations in the neural network; S2, uses the FPGA module to perform pipeline preprocessing of image data collected by the image sensor and control of the RRAM module; S3, uses RRAM modules to perform analog domain multiplication and accumulation calculations on the preprocessed image data in the convolutional layer and fully connected layer of the neural network.

[0059] When performing image recognition, the collected image is pre-processed by the FPGA module and then input into the neural network model. Accurate classification and recognition are achieved through collaborative calculations by the RRAM module and the MPU module, and the final recognition results are returned to the host computer or execution end.

[0060] In the embodiment, the multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture and method provided by the embodiment of the present invention are used to perform infrared gesture recognition. In order to achieve efficient execution of infrared gesture recognition tasks, the neural network model adopted is selected as a lightweight convolutional neural network structure, which is adapted to the resource-constrained characteristics of the low-power edge computing platform.

[0061] Specifically, if Figure 7 As shown in the figure, the neural network model consists of two convolutional layers, two pooling layers, two fully connected layers, and multiple nonlinear activation and normalization operations. The network input is a 64×64 pixel single-channel infrared image, and the output is the probability value of three types of gestures (corresponding to the "rising", "hovering", and "falling" states respectively). Figure 8 Shown are three representative gesture images with high discrimination, representing the recognition results of "ascending," "hovering," and "landing." These gestures exhibit stable thermal distribution characteristics and morphological differences in infrared images, which is conducive to improving the training convergence efficiency and final recognition accuracy of the neural network model.

[0062] In this embodiment, the computational tasks of each layer of the neural network are collaboratively completed by three types of computing units based on their functional characteristics: the RRAM array is responsible for executing large-scale linear multiplication-addition calculations such as convolutional layers and fully connected layers, fully leveraging its high parallelism and low power consumption advantages; the MPU is primarily responsible for nonlinear and logic control operations such as normalization, activation functions, flattening, and Softmax, and participates in light-load computing tasks; the FPGA, as the system scheduling and control core, is responsible for image data preprocessing, driving and controlling the RRAM array, and scheduling and managing data between modules, thereby building a heterogeneous neural network computing system with a reasonable division of labor and efficient collaboration. By transferring the weight calculations of the convolutional and fully connected layers to the RRAM, and nonlinear and control operations to the MPU, while the FPGA coordinates image preprocessing and control operations, the highly energy-efficient operation of neural networks on resource-constrained edge devices is effectively achieved. This is particularly suitable for scenarios such as real-time infrared image recognition, and has significant practicality and promotional value.

[0063] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture, characterized by: include: The MPU module, FPGA module and RRAM module form a tightly coupled collaborative relationship through on-board or on-chip high-speed interconnection; The MPU module is responsible for image sensor control, system coordination management and nonlinear domain calculation in neural networks; The FPGA module is responsible for pipeline preprocessing of image data collected by the image sensor and control of the RRAM module; The RRAM module is responsible for the analog domain multiplication and accumulation calculation of the pre-processed image data in the convolutional layer and the fully connected layer in the neural network; The analog domain multiplication and accumulation calculation results of the convolutional layer and fully connected layer of the RRAM module are transmitted to the MPU module through the FPGA module. After combining with the nonlinear domain calculation of the MPU module, the inference results are output to complete the image recognition processing.

2. The multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture according to claim 1 is characterized in that: The FPGA module includes an image processing module, a memristor control module and a data transmission module; The image processing module is used to perform preprocessing operations including synchronization, non-uniformity correction, median filtering and scale transformation on the image acquired by the image sensor; The data transmission module is used to rearrange the pre-processed image data stream according to the format requirements of the convolutional neural network and then write it into the RRAM module; The memristor control module is used to realize weight writing, reading and matrix multiplication operation control of the RRAM module by parallel controlling the digital-to-analog converter and the analog-to-digital converter.

3. The multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture according to claim 1 or 2, characterized in that: In the FPGA module, an RJ45 interface is configured for inputting the processed image data into the RRAM module to perform analog domain multiplication and accumulation calculations, an SPI interface is configured for high-speed instruction and data interaction with the MPU module, and a JTAG interface is configured for firmware burning, debugging and online updating with the MPU module.

4. The multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture according to claim 1 is characterized in that: The RRAM module includes several RRAM arrays, adopts a 1T1R cross structure, and each intersection unit is composed of a resistive random access memory and a gate transistor connected in series.

5. The multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture according to claim 1 or 4, characterized in that: The RRAM module also includes a state control and detection module for monitoring key parameters of the RRAM array, including conductivity drift, write stability, and power status, and uploading the monitoring data to the MPU module in real time via an Ethernet communication link.

6. The multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture according to claim 5 is characterized in that: In the RRAM module, the weight matrix of the neural network model is directly mapped to the conductance value of each resistive random access memory cell in the RRAM array. The corresponding input stimulus is applied as an analog voltage to the corresponding SL port, and the RRAM array performs analog domain parallel multiplication and accumulation operations of the convolutional layer or fully connected layer.

7. The multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture according to claim 1 is characterized in that: The MPU module includes a sensor control module, a system collaborative management module and a nonlinear domain calculation module; The sensor control module is used to control the image sensor parameter configuration and receive the image sensor output signal through the video parallel data interface to perform synchronous status monitoring and subsequent processing; The system collaborative management module is used to dynamically update the network structure or weight distribution strategy during operation, accurately synchronize the quantized weights of the neural network to the FPGA and RRAM based on the structural mapping relationship, and synchronously coordinate the FPGA and RRAM states for state monitoring and switching; The nonlinear domain calculation module is used to complete the nonlinear activation function operation and finally complete the classification output or upload the inference result to the main control system.

8. The multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture according to claim 7 is characterized in that: The image sensor output signals include pixel clock, line synchronization, frame synchronization and image data.

9. A multi-hardware collaborative sensing, storage and computing integrated intelligent image processing method, implemented using the multi-hardware collaborative sensing, storage and computing integrated intelligent image processing architecture according to any one of claims 1 to 8, characterized in that: The following steps are involved: Use MPU modules for image sensor control, system collaborative management, and nonlinear domain calculations in neural networks; Use FPGA modules to perform pipeline preprocessing of image data collected by image sensors and control of RRAM modules; The RRAM module is used to perform analog domain multiplication and accumulation calculations on the preprocessed image data in the convolutional layer and fully connected layer of the neural network.

10. The multi-hardware collaborative sensing, storage and computing integrated intelligent image processing method according to claim 9, characterized in that: When performing image recognition, the collected image is pre-processed by the FPGA module and then input into the neural network model. Accurate classification and recognition are achieved through collaborative calculations by the RRAM module and the MPU module, and the final recognition results are returned to the host computer or execution end.

Images