An image processing data analysis platform based on a memristor-based memory and computing integrated architecture

Abstract

The present invention discloses an image processing data analysis platform based on a memristor-based in-memory computing architecture. The present invention relates to the technical field of data recognition and includes an XC7Z020 FPGA control module, which is respectively connected to an OV5640 camera image acquisition module, a memristor array computing module, and an Internet of Things module. This image processing data analysis platform based on a memristor-based in-memory computing architecture significantly improves the speed, energy efficiency, and reliability of image processing through the deep integration of the memristor-based in-memory computing architecture and dynamic optimization technology. It eliminates the data transfer bottleneck of the traditional von Neumann architecture, improves the processing speed compared with the traditional GPU solution, and reduces the system energy consumption at the same time. By adjusting the weight allocation and sharding processing strategy through real-time feature feedback, the recognition accuracy in complex industrial scenarios is improved, and the throughput is also broken through, meeting the real-time processing requirements of high-resolution images.

Description

Technical Field

[0001] The present invention relates to the technical field of data recognition, and particularly to an image processing data analysis platform based on a memristor computing-in-memory architecture. Background Art

[0002] With the accelerating transformation of global manufacturing towards intelligence and automation, computer vision technology, as the core means of information perception and decision-making in industrial scenarios, has been widely applied in fields such as intelligent detection, precise sorting, and robot navigation. However, current mainstream image processing systems are mostly based on traditional CPU or GPU architectures. The von Neumann architecture with separate computing and storage leads to frequent data migration, resulting in a serious "memory wall" bottleneck. Taking real-time defect detection on an industrial production line as an example, traditional systems need to transfer a large amount of image data from sensors to computing units, process it through a multi-layer convolutional neural network, and then return control instructions. This process not only consumes a large amount of time, usually with a single-frame processing delay exceeding 10 ms, but also the energy consumption caused by data transfer accounts for more than 60% of the total power consumption. In addition, complex environmental factors such as light changes and mechanical vibrations in industrial scenarios lead to a significant increase in image noise. Traditional fixed-weight convolution models are prone to misidentifications due to feature matching deviations, with a typical misjudgment rate > 5%, seriously restricting the production line efficiency and quality control level. Existing solutions such as using dedicated ASIC chips can improve the computing speed, but their hardware-cured characteristics are difficult to adapt to the dynamically changing industrial requirements, and the development cost is high, making it difficult to be deployed on a large scale.

[0003] In this context, the memristor computing-in-memory architecture shows breakthrough potential. As a new type of device with non-volatile resistive memory characteristics, a memristor can directly represent weight parameters through conductance values, and achieve a parallel processing mode of "storage is computing" at the physical level. In theory, a memristor array can complete data storage and matrix multiplication and addition operations in the same hardware, eliminating the data transfer overhead in traditional architectures, thereby significantly reducing latency and energy consumption. However, existing memristor-based image processing systems still face three key challenges: First, static weight configuration is difficult to adapt to the dynamic changes of image features in industrial scenarios, resulting in insufficient model generalization ability; second, the physical scale of a single memristor array is limited and cannot meet the parallel processing requirements of high-resolution images, such as 4K industrial cameras; third, the inherent noise sensitivity of analog computing may cause result drift and is difficult to meet industrial-level reliability requirements, such as a misidentification rate < 0.1%. Existing technologies attempt to improve by stacking multiple memristor arrays or introducing redundant verification modules, but these solutions often lead to a sharp increase in system complexity and fail to fundamentally solve the balance problem between computing efficiency and accuracy. Summary of the Invention

[0004] (I) Technical Problems to be Solved

[0005] In view of the deficiencies of the prior art, the present invention provides an image processing data analysis platform based on a memristor computing-in-memory architecture, which solves the problem of how to achieve high-speed processing, low-power operation, and high-reliability recognition of image data in industrial scenarios through the memristor computing-in-memory architecture.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention is realized through the following technical solutions: An image processing data analysis platform based on a memristor computing-in-memory architecture, comprising:

[0008] The XC7Z020 FPGA control module is respectively connected to the OV5640 camera image acquisition module, the memristor array computing module, and the Internet of Things module; it should be further noted that the platform of the present invention takes the XC7Z020 FPGA control module as the core, and realizes the complete process of image acquisition, processing, calculation, and mechanical control through the cooperation of hardware interfaces and software logics.

[0009] The OV5640 camera image acquisition module is used to acquire raw image data and transmit it to the XC7Z020 FPGA control module; it should be further noted that the OV5640 camera image acquisition module is connected to the PS end of the FPGA through the I2C bus to complete the initialization configuration of the camera, including resolution and frame rate settings, and transmits the raw image data to the DDR3 buffer area of the FPGA through the DCMI interface.

[0010] The memristor array computing module includes a D / A converter, a memristor array, a transimpedance amplifier, and an A / D converter, and is used to receive the image data processed by the XC7Z020 FPGA control module, complete the convolution operation through parallel analog computing, and return the result to the XC7Z020 FPGA control module; it should be further noted that the memristor array computing module is composed of a multi-channel D / A converter, a memristor array, a transimpedance amplifier, and an A / D converter. The PL end of the FPGA distributes the preprocessed image data to the D / A converter, converts it into an analog voltage signal and loads it onto the bit line (BLx) of the memristor array. After the convolution operation is completed, the current signal of SLx is converted into a voltage signal by the transimpedance amplifier, and then returned to the FPGA by the A / D converter.

[0011] The Internet of Things module receives instructions from the XC7Z020 FPGA control module through the CAN bus and executes predefined actions; it should be further noted that the Internet of Things module receives instructions from the FPGA through the CAN bus and drives the servo motor and pneumatic gripper to perform sorting, positioning, or obstacle avoidance operations.

[0012] The platform also includes a Dynamic Resource Allocation Module (DRAM). The real-time task scheduling unit is integrated through the XC7Z020 FPGA control module, which is used to dynamically allocate computing tasks to the FPGA logic unit or the memristor array computing module according to the complexity of the input image.

[0013] The real-time task scheduling unit generates task priorities by analyzing the spatial frequency, edge density, and gray-scale distribution of the image, and allocates the convolution operations in high-complexity regions to the memristor array computing module for parallel simulation computing, while retaining the low-complexity processing tasks to be executed in the FPGA logic unit.

[0014] Among them, the weights of the memristor array are pre-written in the form of conductance and loaded onto the array nodes through voltage signals to achieve the memory-computation integrated convolution operation. It should be further noted that the weights of the memristor array are pre-written in the form of conductance, and the computing resources are dynamically scheduled according to the image complexity through the dynamic resource allocation mechanism (DRAM) to achieve efficient processing of memory-computation integration.

[0015] Preferably, the XC7Z020 FPGA control module is used to segment, compress, and perform gray-scale processing on the original image to generate preprocessed data; and transmit the preprocessed data to the memristor array computing module, and control the Internet of Things module to execute actions. It should be further noted that the XC7Z020 FPGA control module segments, compresses, and performs gray-scale processing on the original image collected by the OV5640 camera. Image segmentation adopts a dynamic block strategy, and the image is divided into sub-regions of different sizes according to the texture complexity, including: high-texture regions are sliced into 64×64 pieces, low-texture regions are sliced into 256×256 pieces, and the edges of the slices are processed by bilinear interpolation. The compression process adopts differential pulse code modulation (DPCM) and Huffman coding, with a compression ratio of 2:1, and the hardware logic unit achieves a throughput of 1.5 Gbps. Gray-scale processing is performed through the brightness formula recommended by ITU: Y = 0.299R + 0.587G + 0.114B; the RGB image is converted into 8-bit gray-scale data, and the DSP Slices of the FPGA complete the multiplication and addition operations in parallel, with a single-frame processing delay of 1.08 ms. The preprocessed data is transmitted to the memristor array computing module through the LVDS interface. At the same time, the FPGA generates a manipulator action instruction according to the recognition result, encapsulates it into a data frame containing coordinate and speed parameters through the CAN bus protocol, and drives the servo motor to complete the operation.

[0016] Preferably, the memristor array computing module includes a dynamic weight synchronization network DWS-Net, that is: the XC7Z020 FPGA control module embeds a lightweight feature extraction unit to analyze the spatial frequency and texture distribution of the input image in real time and generate a feature vector; based on the feature vector, the conductance values of different nodes in the memristor array are dynamically adjusted through the D / A converter to achieve adaptive allocation of convolution kernel weights; the dynamic weight synchronization network optimizes weight matching through a closed-loop feedback mechanism to reduce redundant computing energy consumption. It should be further noted that the dynamic weight synchronization network DWS-Net dynamically adjusts the weight allocation of the memristor array through real-time image feature feedback. The PS side of the FPGA embeds a lightweight feature extraction unit, which uses FFT to calculate the high-frequency energy ratio and Sobel operator to count the edge density, and generates a 24-bit feature vector. According to the feature vector, the D / A converter generates voltage pulses with specific amplitudes, where +3V is used for the SET operation and -3V is used for the RESET operation to dynamically regulate the conductance values of the memristor nodes. It includes: increasing the conductance value of the edge detection kernel when the high-frequency energy ratio > 60%, and reducing the weight of the smoothing filter kernel in low-contrast scenarios. The closed-loop feedback mechanism combines the results of bimodal verification. If the confidence level is lower than the threshold, the conductance value is corrected again. The weight adjustment realizes the weighted summation of current signals through Kirchhoff's current law, and the single adjustment delay < 0.5ms, and the misrecognition rate is reduced.

[0017] Preferably, the memristor array computing module integrates a parallel sharding convolution accelerator PSCA, that is: the XC7Z020 FPGA control module divides the input image into N×N independent sub-regions, regarded as shards, and each shard is assigned to an independent sub-array of the memristor array; each sub-array completes local convolution operations through parallel analog computing and synchronously outputs current signals through the multiplexing technology of a transimpedance amplifier; the A / D converter converts the analog signals of multiple sub-arrays into digital signals, and the XC7Z020 FPGA control module performs global feature fusion. It should be further noted that the parallel sharding convolution accelerator PSCA divides the high-resolution image into N×N independent sub-regions, including 4×4 and 8×8 shards, and each shard is assigned to an independent sub-array of the memristor array. The FPGA distributes the shard data to multiple D / A converters through the AXI-Stream interface, converts it into a 0-1V analog voltage and loads it onto the BLx line. Each sub-array completes the analog convolution operation in parallel. The current signal output by the SLx is converted into a voltage signal through the gain of a transimpedance amplifier, and then switched to the A / D converter for synchronous sampling through a 32-channel analog switch ADG732. The FPGA aligns the shard results according to the time stamp and adopts weighted averaging: the weight of high-complexity shards is 0.7 or the maximum value fusion algorithm to generate a global feature map. In the shard mode, the processing speed of 4K images is compressed from 12ms to 2.4ms, and the throughput is increased by 5 times.

[0018] Preferably, the XC7Z020 FPGA control module deploys a Dual-Mode Verification Mechanism (DMVM): constructing a memristor array in the first mode to simulate the convolutional operation path, and an FPGA digital logic verification path in the second mode; comparing the calculation results of the two modes through a confidence evaluation module, and triggering secondary calculation or alarm if the difference exceeds a preset threshold; the Dual-Mode Verification Mechanism controls the mis-identification rate below 0.05% in a noise interference environment. It should be further noted that the Dual-Mode Verification Mechanism (DMVM) constructs a dual-path of analog calculation and digital verification. The analog path completes the convolutional operation by the memristor array, and the digital path performs fixed-point multiplication and addition operations through the FPGA logic unit. The confidence evaluation module calculates the Euclidean distance of the results of the two paths, and triggers secondary calculation or alarm if the difference exceeds the dynamic threshold, where the initial value is 0.1 and is adaptively adjusted according to historical data. It includes: under the noise interference of SNR = 15dB, the mis-identification rate of the analog path is 0.12%, and it drops to 0.03% after digital verification. The FPGA outputs high and low level alarm signals through GPIO pins, drives the LED or buzzer to prompt anomalies, and records the fault type in the Block RAM for subsequent diagnosis.

[0019] Preferably, the weight adjustment process of the Dynamic Weight Synchronization Network (DWS-Net) includes: generating a voltage pulse signal through the D / A converter to regulate the initial conductance value of the memristor array; dynamically correcting the conductance distribution according to real-time image features to match the spatial frequency and gray gradient of the input data; the weight adjustment process realizes the weighted summation of current signals through Kirchhoff's current law to complete the convolutional operation. It should be further noted that the weight adjustment process of DWS-Net includes voltage pulse regulation and closed-loop calibration. In the initialization stage, the D / A converter writes the conductance value into the memristor array through ±3V pulses, and immediately reads the SLx current after writing to reverse the actual conductance value, and triggers re-writing when the error exceeds 5%. During real-time processing, the FPGA dynamically corrects the conductance distribution according to the spatial frequency and gray gradient.

[0020] Preferably, the sharding process of the Parallel Sharding Convolution Accelerator (PSCA) includes: dividing the high-resolution image into 4×4 or 8×8 shards, and each shard corresponds to an independent memristor sub-array; the input voltage of each sub-array is generated by the D / A converter based on the shard gray value, and synchronously outputs the current signal through a transimpedance amplifier; the A / D converter synchronously samples the multi-path sub-array signals to ensure the timing consistency of the global convolutional result. It should be further noted that the sharding process divides the high-resolution image into 4×4 or 8×8 shards, and each shard is bound to an independent sub-array. The D / A converter generates a 0-1V voltage signal based on the shard gray value and loads it onto the BLx line, and the transimpedance amplifier synchronously outputs multi-path sub-array current signals. The A / D converter samples at a rate of 100MSPS, and the FPGA integrates the results through the timestamp matching and weighted fusion algorithm.

[0021] Preferably, the circuit structure of the memristor array calculation module includes: a multi-channel D / A converter and a transimpedance amplifier, which are respectively connected to the bit line BLx and the source line SLx of the memristor array;

[0022] The D / A converter regulates the conductance value of the memristor through a voltage pulse signal, and the transimpedance amplifier converts a tiny current signal into a voltage signal;

[0023] After being filtered by a high-speed operational amplifier, the A / D converter converts the analog signal into a digital signal and transmits it back to the XC7Z020 FPGA control module.

[0024] It should be further noted that the memristor array circuit includes a multi-channel D / A converter, a transimpedance amplifier and an A / D converter. The D / A converter AD9744 receives the FPGA digital signal through the LVDS interface, generates an analog voltage and loads it onto the BLx line to regulate the conductance value of the memristor. The transimpedance amplifier OPA657 converts the μA-level current of SLx into a mV-level voltage. After eliminating noise through the second-order low-pass filter LMH6521, it is sampled by the A / D converter AD9434 into a 12-bit digital signal and transmitted back to the FPGA. The power management chip LM27762 provides ±5V voltage for the D / A converter to ensure the stability of the SET / RESET operation. The multi-channel analog switch ADG732 switches the signal path according to the shard index, and the timing deviation <1ns.

[0025] Preferably, the process of the Internet of Things module receiving instructions through the CAN bus includes: the XC7Z020 FPGA control module generates an action instruction according to the fully connected result of the convolutional neural network; the instruction is transmitted to the robotic arm drive circuit through the CAN_H and CAN_L signals to perform sorting, positioning or obstacle avoidance operations. It should be further noted that the Internet of Things module receives the instruction frame generated by the FPGA through the CAN bus, standard extended frame ID: 0x18FF0000. The FPGA encapsulates the fully connected result of the convolutional neural network into an 8-byte data field containing action type, coordinates, and speed parameters, and appends a CRC-16 check code. The CAN transceiver SN65HVD230 converts the TTL signal into differential levels CAN_H / CAN_L and transmits it to the robotic arm drive circuit. The servo motor converts the coordinates into joint angles according to the inverse kinematics model, and the PWM signal controls the rotation with an accuracy of ±0.1°. In the obstacle avoidance scenario, the ToF sensor VL53L1X detects obstacles in real time, the FPGA plans the path through the A* algorithm, the response time <10ms, and the gripper pressure sensor feeds back data to form a closed-loop control.

[0026] Preferably, the dynamic resource allocation mechanism optimizes the platform energy consumption through a load balancing algorithm, including: when it is detected that the complexity of consecutive multiple frames of images is lower than a preset threshold, the power supply of some memristor sub-arrays is turned off to reduce power consumption; in a high-load scenario, the redundant sub-array of the parallel sharding convolutional accelerator (PSCA) is enabled for hyper-thread acceleration. It should be further noted that the dynamic resource allocation mechanism (DRAM) optimizes the energy consumption through a load balancing algorithm. The FPGA calculates the image complexity score in real time, including spatial frequency, edge density, and gray entropy weighting. When the load is low, that is, when the score < 40, 50% of the sub-arrays are turned off, and the power consumption is reduced from 5.2W to 3.1W; when the load is high, that is, when the score > 80, the redundant sub-array is enabled, and the hyper-thread acceleration throughput reaches 480 frames per second. The power management chip TPS22965 controls the power supply of the sub-array, and the sleep power consumption is only 5μW. The closed-loop regulation dynamically adjusts the voltage according to the real-time power consumption monitored by MAX44284. For example, when the power consumption exceeds 5.5W, the voltage is reduced to 0.8V to ensure that the energy efficiency ratio reaches 96 GOPS / W. The historical learning model optimizes the sharding and merging strategy, compensating for the loss of calculation accuracy to < 0.5%.

[0027] (III) Beneficial effects

[0028] The present invention provides an image processing data analysis platform based on a memristor memory-computation integrated architecture, having the following beneficial effects:

[0029] (I). This image processing data analysis platform based on a memristor memory-computation integrated architecture significantly improves the speed, energy efficiency, and reliability of image processing through the deep integration of the memristor memory-computation integrated architecture and dynamic optimization technology. First, the memristor array implements parallel convolution operations in an analog manner, eliminating the data transfer bottleneck of the traditional von Neumann architecture. The processing speed is improved compared with the traditional GPU solution, and at the same time, the system energy consumption is reduced. The dynamic weight synchronization network and the parallel sharding convolutional accelerator further optimize the calculation efficiency. By adjusting the weight distribution and sharding processing strategy in real time through feature feedback, the recognition accuracy in complex industrial scenarios is improved, and the throughput is also broken through, meeting the real-time processing requirements of high-resolution images.

[0030] (II). This image processing data analysis platform based on a memristor memory-computation integrated architecture, the dual-modal verification mechanism and the dynamic resource allocation mechanism enhance the robustness and adaptability of the system. The DMVM reduces the misrecognition rate through cross-verification of the analog and digital paths, improving the industrial-level reliability; the DRAM dynamically starts and stops the sub-array and redundant resources according to the load, reducing the power consumption at low load and improving the throughput through hyper-thread acceleration at high load, taking into account both energy efficiency and flexibility. The overall solution shows excellent performance in scenarios such as semiconductor detection and intelligent sorting, which is conducive to promoting the intelligent upgrade of the manufacturing industry. Description of the drawings

[0031] Figure 1 Schematic diagram of the overall framework of the present invention;

[0032] Figure 2 Schematic diagram of the internal process of the memristor of the present invention. Specific implementation manners

[0033] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] Please refer to Figure 1 and Figure 2 The present invention provides a technical solution: An image processing data analysis platform based on a memristor memory-computation integrated architecture, including:

[0035] An XC7Z020 FPGA control module, which is respectively connected to an OV5640 camera image acquisition module, a memristor array calculation module, and an Internet of Things module;

[0036] The OV5640 camera image acquisition module is used to acquire original image data and transmit it to the XC7Z020 FPGA control module; the OV5640 camera image acquisition module is connected to the PS end of the FPGA through the I2C bus for camera initialization configuration, including resolution and frame rate settings. The original image data collected by the camera is transmitted to the PL end of the FPGA through the D0-D7 data lines of the DCMI to achieve high-speed data reception. The process is as follows: First, image data acquisition is performed. The FPGA configures the working mode of the OV5640 camera through the I2C bus: 1080P@30fps, and triggers image acquisition. The original image data is transmitted to the DDR3 buffer area at the PS end of the FPGA in the YUV format to complete frame synchronization and data alignment; then image preprocessing is performed. The PS end of the FPGA converts the YUV data into an 8-bit grayscale image through the weighted average method in the RGB to grayscale algorithm to reduce the subsequent calculation complexity. Subsequently, a median filtering algorithm, that is, a 3×3 window is used to remove salt-and-pepper noise and improve the image quality. Then, according to the preset ROI region, the image is segmented into multiple sub-regions, that is, 512×512 pixel blocks, for subsequent parallel processing.

[0037] The memristor array computing module includes a D / A converter, a memristor array, a transimpedance amplifier, and an A / D converter. It is used to receive the image data processed by the XC7Z020 FPGA control module, complete the convolution operation through parallel analog computing, and return the result to the XC7Z020 FPGA control module. The memristor array computing module is connected to the D / A converter and the A / D converter through the GPIO pins of the PL side of the FPGA. Among them, the D / A converter receives the digital signal processed by the FPGA, converts it into an analog voltage signal, and loads it onto the bit line BLx of the memristor array to complete the weight writing and input data loading. The transimpedance amplifier TIA converts the tiny current signal on the array source line SLx into a voltage signal, which is filtered and then returned to the FPGA by the A / D converter.

[0038] The convolution kernel weights are determined through offline training and are pre-written into the memristor array in the form of conductance values. The specific process is as follows: The FPGA generates voltage pulse signals with specific amplitudes through the D / A converter: AD9744, including +3V for the SET operation and -3V for the RESET operation, which are applied to the cross nodes of BLx and SLx of the memristor array to regulate its conductance value to the target range, that is, 10 μS to 100 μS. After the writing is completed, the accuracy of the conductance value is verified by reading the current signal to ensure that the weights are consistent with the training model.

[0039] Perform parallel analog convolution operations, including: The pre-processed image grayscale values (0 - 255) are converted into analog voltage signals (0 - 1V) through the D / A converter and loaded onto the BLx line of the memristor array; According to Kirchhoff's current law, the output current of each SLx line is the sum of the products of the BLx voltage and the corresponding conductance values, that is , to achieve the analog calculation of the convolution operation. The current signal of SLx is converted into a voltage signal by the transimpedance amplifier OPA657 with a gain set to 1000 times, and then sampled into a 12-bit digital signal by the high-speed A / D converter AD9434 and transmitted to the PL side of the FPGA.

[0040] The Internet of Things module receives instructions from the XC7Z020 FPGA control module through the CAN bus and executes predefined actions; that is: The Internet of Things module is connected to the PS side of the FPGA through the CAN bus, receives the recognition result instructions, including sorting coordinates and action types, and drives the robotic arm to execute operations; including: The PL side of the FPGA performs pooling and fully connected operations on the convolution results returned by the memristor array to generate the final recognition result. Among them, the pooling performed is max pooling with a window of 2×2, and the fully connected is based on the pre-trained weight matrix. The final recognition result includes the defect category and coordinate information;

[0041] The recognition result is encapsulated as an instruction frame through the CAN bus protocol and sent to the robotic arm drive circuit by the CAN controller of the FPGA. Among them, the standard frame ID is 0x18FF0000, and the CAN controller is integrated on the PS side. The robotic arm performs sorting, positioning, or obstacle avoidance operations according to the instructions, and the response delay is less than 5 ms. Among them, the sorting operation includes the action of the pneumatic gripper, the positioning operation includes the rotation angle of the servo motor, and the obstacle avoidance operation is a path planning algorithm.

[0042] The platform also includes a Dynamic Resource Allocation Module (DRAM). The XC7Z020 FPGA control module integrates a real-time task scheduling unit, which is used to dynamically allocate computing tasks to the FPGA logic unit or the memristor array computing module according to the complexity of the input image. The real-time task scheduling unit generates task priorities by analyzing the spatial frequency, edge density, and gray distribution of the image, and allocates the convolution operations in high-complexity regions to the memristor array computing module for parallel analog computing. The low-complexity processing tasks are retained in the FPGA logic unit for execution.

[0043] The implementation of the dynamic resource allocation mechanism DRAM includes: task scheduling unit design and dynamic power management. Among them, the task scheduling unit design process includes:

[0044] The PS side of the FPGA embeds a lightweight feature extraction algorithm to calculate the spatial frequency and edge density of the image in real time, and generates a task priority score, including 0-100 points. Among them, the spatial frequency is the proportion of the frequency domain energy through Fourier transform, and the edge density is the gradient amplitude statistics through the Sobel operator.

[0045] Task allocation strategy:

[0046] High-complexity regions, that is, scores > 70: are allocated to the memristor array computing module for parallel analog convolution, taking advantage of its high-speed and low-power characteristics.

[0047] Low-complexity tasks, that is, scores ≤ 70: are executed by the PL side of the FPGA logic unit for digital convolution to avoid occupying high-performance resources.

[0048] Dynamic power management includes low-load mode and high-load mode. Among them, the low-load mode: when the complexity scores of 10 consecutive frames of images are all lower than 50, the FPGA turns off the power supply of some memristor sub-arrays through the power management chip TPS65218, and only retains 1 / 4 of the array for operation, and the system power consumption is reduced to 3.8 W. High-load mode: When a burst of high-complexity tasks is detected, that is, scores > 90, redundant sub-arrays are enabled, that is, through multiplexer switches for switching, and parallel processing of sharded data, and the single-frame processing delay is compressed to 0.8 ms.

[0049] Among them, the weights of the memristor array are pre-written in the form of conductance and loaded onto the array nodes through voltage signals to achieve the convolutional operation of the integration of storage and computing.

[0050] The XC7Z020 FPGA control module is used to segment, compress, and grayscale the original image to generate preprocessed data; and transfer the preprocessed data to the memristor array computing module and control the manipulator to execute actions.

[0051] It should be further noted that in the specific implementation process, image segmentation includes: the PL side of the XC7Z020 FPGA, that is, the Programmable Logic configures the image segmentation hardware acceleration module. Through the predefined ROI, that is, the Region of Interest algorithm, the original image collected by the OV5640 camera, such as with a resolution of 1920×1080, is divided into multiple sub-regions. The block size is dynamically adjusted according to the complexity of the image content. For example, in the industrial defect detection scenario, for high-texture regions, that is, solder joints and edges, small blocks of 64×64 pixels are used, and for low-texture background regions, large blocks of 256×256 pixels are used to balance the computational resources and accuracy requirements. The bilinear interpolation algorithm is used to process the block edge pixels to avoid image distortion caused by segmentation. The parallel pipeline design of the FPGA allows multiple blocks to be processed simultaneously, and the single-frame segmentation delay is less than 0.2 ms at 1080P resolution.

[0052] Image compression uses differential pulse code modulation (DPCM) technology to perform predictive coding on the grayscale image data. The specific steps are as follows: the current pixel value is predicted based on the left neighbor pixel to generate a residual signal. The residual is mapped to the 8-bit signed integer range (-128~127) to reduce the data volume. The data is further compressed through Huffman coding, and the compression ratio can reach 2:1 (measured average value). The PL side of the FPGA integrates a dedicated compression logic unit, such as a LUT-based encoder, and the compression throughput reaches 1.5 Gbps to ensure real-time performance.

[0053] Grayscale processing includes: RGB to grayscale formula: using the luminance formula recommended by the International Telecommunication Union (ITU): Y = 0.299R + 0.587G + 0.114B; where R, G, and B are the three-channel values of the original pixel, and Y is the grayscale value. Parallel multiplication and addition operations are implemented through the DSP Slice of the FPGA. The single-pixel processing cycle is 1 clock cycle. That is, at a main frequency of 100 MHz, the grayscaling of a single-frame 1080P image takes 1.08 ms. A real-time median filter with a 3×3 window is embedded after grayscaling to eliminate sensor noise and improve the accuracy of subsequent convolutional operations.

[0054] The preprocessed image data is transmitted to the D / A converter AD9744 of the memristor array computing module through the GPIO pins on the PL side of the FPGA, such as the LVDS differential pair of Bank34. The 8-bit grayscale value (0 - 255) is linearly scaled to a 14-bit digital signal (0 - 16383) to match the input precision of the D / A converter. The D / A converter converts the digital signal into an analog voltage signal (0 - 1V) and loads it onto the bit lines (BLx) of the memristor array. The FPGA generates a synchronous clock signal (CLK+ / -) to control the sampling timing of the D / A converter, ensuring strict synchronization between data loading and convolution operations.

[0055] In the system initialization stage, the trained convolution kernel weights are written into the memristor array in the form of conductance values through the D / A converter. A +3V pulse is used to increase the conductance (SET operation), and a -3V pulse is used to decrease the conductance (RESET operation), with a stable voltage source provided by a charge pump chip (LM27762). In real-time processing, combined with the dynamic weight synchronization network DWS-Net described below, the conductance values are dynamically corrected according to image features to optimize the convolution kernel matching.

[0056] Control the robotic arm to execute actions: The PL side of the FPGA performs max pooling (window 2×2) and fully connected operations on the convolution results returned by the memristor array. The pre-trained fully connected layer weights are stored in the Block RAM of the FPGA, and matrix multiplication is completed through a parallel multiplier-adder (DSP48E1 Slice) to output classification probabilities or coordinate information. The recognition results (such as defect categories, target coordinates) are encapsulated into 32-bit data packets (including check bits) and transmitted to the PS side (Processing System) of the FPGA.

[0057] The robotic arm control instruction transmission uses the standard CAN 2.0B extended frame format, with a frame ID of 0x18FF0000. The data field contains the action type (1 byte), coordinates X / Y (2 bytes each), and speed parameter (1 byte). The PS side of the FPGA integrates a CAN controller (IP Core), supporting a communication rate of 1Mbps, and the instruction transmission delay is less than 0.1ms.

[0058] The Internet of Things module analyzes the CAN instructions and drives the servo motor to rotate to the target angle (accuracy ±0.1°) through PWM signals. In complex scenarios, such as multi-target sorting, the FPGA pre-sets the A* algorithm to generate the optimal path to avoid collisions of the robotic arm. A force sensor and a vision feedback module are installed at the end of the robotic arm to transmit the execution status (such as clamping force, position deviation) back to the FPGA in real time. If an abnormality is detected (such as clamping failure), the FPGA triggers a recalculation process to update the robotic arm instructions to ensure the reliability of the operation.

[0059] This implementation process fully demonstrates the advantages of the memristor-based memory and computing integrated architecture in terms of speed, energy consumption, and reliability through hardware-algorithm co-optimization, dynamic resource allocation, and closed-loop control mechanisms.

[0060] The memristor array computing module includes a dynamic weight synchronization network (DWS-Net). The DWS-Net includes a lightweight feature extraction unit embedded in the XC7Z020 FPGA control module, which analyzes the spatial frequency and texture distribution of the input image in real time to generate a feature vector. Based on the feature vector, the conductance values of different nodes in the memristor array are dynamically adjusted through a D / A converter to achieve adaptive allocation of convolution kernel weights. The DWS-Net optimizes weight matching through a closed-loop feedback mechanism to reduce redundant computing energy consumption.

[0061] It should be further noted that in the specific implementation process, the architecture design of the DWS-Net aims to dynamically adjust the weight allocation of the memristor array through real-time image feature feedback to solve the problem of insufficient generalization ability caused by traditional fixed weights. Its implementation process includes the following key steps:

[0062] Design and implementation of the lightweight feature extraction unit: The input image is decomposed in the frequency domain using the Fast Fourier Transform (FFT), and the high-frequency energy ratio is calculated, such as the proportion of components exceeding 50Hz in the frequency domain, to characterize the image detail complexity.

[0063] Formula: ; where F(u,v) is the two-dimensional Fourier transform result of the image.

[0064] Texture features are extracted based on the Gray Level Co-occurrence Matrix (GLCM), including parameters such as contrast, energy, and correlation, and a texture complexity score is comprehensively generated.

[0065] Contrast calculation: ;

[0066] where P(i,j) is the joint probability of gray levels i and j in the gray level co-occurrence matrix.

[0067] The PL side (Programmable Logic) of the FPGA integrates an FFT IP core (such as Xilinx FFT v9.0), which supports real-time transformation of 512×512 pixel blocks (delay <0.1ms). The gray level co-occurrence matrix is stored in the Block RAM of the FPGA, and parameters such as contrast and energy are calculated in parallel using DSPSlice. The single-frame processing time is 0.3ms. Parameters such as the high-frequency energy ratio (1D), texture contrast (1D), and energy (1D) are normalized to 8-bit integers and combined into a 3D feature vector for subsequent weight adjustment.

[0068] Dynamic adjustment mechanism of the conductance value of the memristor array: The convolutional kernel weights are determined through offline training, and each weight value w ij corresponds to the memristor cross-point BL i , SL j with the target conductance value G ij , and the mapping formula is:

[0069] ; where k is the proportionality coefficient, which is determined by the memristor conductance range, and G minThe minimum conductance value is set to prevent the conductance from returning to zero. This process includes: Step 1: Feature Vector Analysis: The FPGA's PS receives the feature vector and generates conductance adjustment instructions based on preset rules. This includes: If the high-frequency energy ratio is greater than 60%, the image is considered rich in detail, and the weight of the edge detection convolution kernel is increased, thereby increasing the corresponding conductance value. If the texture contrast is less than 30, the image is considered smooth, and the weight of the high-frequency filter convolution kernel is decreased, thereby decreasing the corresponding conductance value. Step 2: Voltage Pulse Generation: A D / A converter (AD9744) generates a voltage pulse signal of a specific amplitude, including SET and RESET operations. The SET operation consists of a +3V pulse (10ns duration) to increase the conductance value. The RESET operation consists of a -3V pulse (10ns duration) to decrease the conductance value. The pulse amplitude and width are dynamically adjusted based on the conductance adjustment range. For example, if the conductance needs to increase by 20%, the pulse amplitude is increased to +3.5V. Step 3: Conductance write verification: Immediately after writing, the SL line current is read through a transimpedance amplifier (TIA) to infer the actual conductance value and compare it with the target value. If the error exceeds 5%, a second write or an alarm is triggered to ensure weight accuracy. A closed-loop feedback mechanism optimizes weight matching: The confidence score output by the dual-modal verification mechanism (DMVM), such as the difference in Euclidean distance, is used. The power consumption of the memristor array is collected in real time (via the MAX44284 current sensor) to calculate the energy consumption per frame. If the confidence score falls below a threshold, such as <90%, the current weight matching is considered insufficient, triggering the feature extraction unit to reanalyze the image and generate new conductance adjustment instructions. If the energy consumption per frame exceeds a preset value, such as >1mJ, the weights are considered redundant and the conductance values of non-critical convolution kernels are automatically reduced. The conductance adjustment rules are optimized based on historical data (such as feature vectors from 10 consecutive frames), for example, by using a linear regression model to predict the optimal weight distribution. A lightweight feature extraction algorithm (FFT and GLCM calculations) is run, occupying 20% of the ARM Cortex-A9 core resources. Conductivity adjustment control logic (occupies 10% of LUT resources). D / A converter interface and transimpedance amplifier signal processing (occupies 15% of GPIO pins). From feature extraction to conductance adjustment, the entire process latency is less than 0.5ms, meeting 30fps real-time processing requirements. Dynamic weight adjustment reduces redundant computation by 40%, lowering overall system power consumption to 3.8W (compared to 4.5W for a static weighting solution). In industrial defect detection scenarios, the false recognition rate has been reduced from 0.1% to 0.04%.

[0070] The memristor array computing module integrates a parallel sliced convolutional accelerator (PSCA), that is: the XC7Z020 FPGA control module divides the input image into N×N independent sub-regions, regarded as slices, and each slice is assigned to an independent sub-array of the memristor array; each sub-array completes local convolutional operations through parallel analog computing, and synchronously outputs current signals through the multiplexing technology of transimpedance amplifiers; the A / D converter converts the analog signals of multiple sub-arrays into digital signals, and the XC7Z020 FPGA control module performs global feature fusion.

[0071] It should be further noted that in the specific implementation process, the parallel sliced convolutional accelerator (PSCA) aims to break through the physical scale limitation of a single array by dividing a high-resolution image into independent sub-regions (slices) and utilizing the parallel computing ability of the memristor sub-arrays, significantly improving the processing speed. Its implementation process includes the following key steps:

[0072] S1. Image slicing strategy and hardware implementation: Dynamically determine the slice size according to the resolution and complexity of the input image. Include: For high-resolution images, such as 3840×2160 pixel images collected by a 4K industrial camera, 8×8 slices are used, with each slice being 480×270 pixels, ensuring the efficient utilization of sub-array computing resources. For high-texture regions (such as solder joints of electronic components), 4×4 slices are used, with each slice being 240×135 pixels, improving the accuracy of local feature extraction.

[0073] Adopt an overlapping slicing strategy (Overlap = 10%) to avoid loss of edge information. Include: Adjacent slices overlap by 24 pixels in the horizontal and vertical directions (480×270 slices), and are smoothly transitioned through an interpolation algorithm (bicubic interpolation).

[0074] The FPGA slicing logic includes: The PL side (Programmable Logic) of the XC7Z020 FPGA integrates an image slicing hardware acceleration module, which is implemented through pixel address mapping and parallel data distribution steps; Pixel address mapping maps the pixel coordinates of the original image to the slice coordinate system, generating a slice index table (stored in Block RAM). Parallel data distribution transfers slice data to the D / A converter interfaces of multiple memristor sub-arrays simultaneously through the AXI-Stream interface of the FPGA.

[0075] Adopt a pipeline architecture, and the single-frame slice processing delay is less than 0.3ms (4K resolution).

[0076] S2. Parallel computing process of the memristor sub-array: The pixel grayscale value (0 - 255) of each slice is converted into an analog voltage signal (0 - 1V) through linear scaling and loaded onto the bit line BLx of the corresponding sub-array by a D / A converter (AD9744); A negative voltage (-1V) is generated by a charge pump chip (LM27762) to represent the negative weight value of the convolution kernel. The conductance values of the memristors in each sub-array are pre-written according to the slice task type (such as edge detection, texture classification) to ensure the independence of the sub-array functions.

[0077] According to Kirchhoff's current law, the output current of the source line SLx of each sub-array is:

[0078] ; where, V BLi is the bit line voltage, and G i is the conductance value of the corresponding memristor node.

[0079] The SLx line of each sub-array is connected to an independent transimpedance amplifier (TIA, such as OPA657) with a gain set to 1000 times to convert the μA-level current signal into a mV-level voltage signal. An analog multiplexer (such as ADG732) is used to select the current sampled sub-array signal to ensure the timing synchronization of the multi-channel output. A high-speed A / D converter (AD9434) synchronously samples the voltage signals of multiple sub-arrays with a 12-bit precision and a sampling rate set to 100MSPS to ensure distortion-free data. The PL end of the FPGA matches the convolution results of each slice through timestamps to eliminate the time offset of the slice processing. A weighted average algorithm is used to fuse the slice results, and the weights are dynamically determined by the slice complexity, including a weight of 0.7 for high-texture slices and 0.3 for low-texture slices.

[0080] S3. Hardware circuit design and performance optimization: The BLx and SLx of different sub-arrays are isolated by metal wires to avoid signal crosstalk, where each sub-array contains 128×128 memristor cross nodes. Each sub-array is independently powered, and the power switch (such as TPS22965) is controlled by the FPGA to turn off the idle sub-arrays under low load, reducing the power consumption by 30%. The ADG732 analog multiplexer is used to support the dynamic switching of 32 SLx signals with a switching delay less than 10ns. A high-speed operational amplifier (LMH6521) is embedded at the output end of the transimpedance amplifier to form a low-pass filter (cut-off frequency 50MHz) to eliminate high-frequency noise.

[0081] The XC7Z020 FPGA control module deploys a dual-mode verification mechanism DMVM: constructs the first-mode memristor array analog convolution operation path and the second-mode FPGA digital logic verification path; the confidence evaluation module compares the calculation results of the two modes, and if the difference exceeds the preset threshold, triggers secondary calculation or alarm; the dual-mode verification mechanism controls the mis-identification rate below 0.05% in a noise interference environment.

[0082] It should be further noted that in the specific implementation process, the dual-modal verification mechanism (DMVM) solves the potential noise interference and precision drift problems in memristor analog computing by constructing cross-verification of heterogeneous computing paths (simulation and digital), ensuring high-reliability recognition in industrial scenarios. Its implementation process includes the following steps:

[0083] T1. Memristor array simulates the convolution operation path: The trained convolution kernel weights are pre-written into the memristor array in the form of conductance values. The edge detection convolution kernel , and the corresponding memristor node conductance values are proportionally mapped (negative weights are represented by negative voltages). The preprocessed image grayscale values are converted into analog voltage signals (0 - 1V) through a D / A converter (AD9744) and loaded onto the bit lines (BLx) of the memristor array. Based on Kirchhoff's law, the source line (SLx) outputs a current that is the sum of the products of the bit line voltages and conductance values, implementing the analog-domain convolution operation. A transimpedance amplifier (TIA) converts the μA-level current signal into a mV-level voltage signal, which is sampled by an A / D converter (AD9434) into a 12-bit digital result and transmitted to the PL side of the FPGA. Combining with a parallel sharding convolution accelerator (PSCA), the image is segmented into sub-regions for parallel processing, and the single-frame analog convolution delay is compressed to 0.8 ms (1080P resolution). A low-pass filter (LMH6521, cut-off frequency 50 MHz) is embedded before the A / D conversion to eliminate high-frequency noise interference and ensure signal stability.

[0084] T2. FPGA digital logic verification path: The PL side of the FPGA integrates a digital convolution IP core, supporting various convolution kernel sizes such as 3×3 and 5×5. Parallel multiply-accumulate operations are implemented through a hardware description language (HDL). After the original image data is grayscaled, it is simultaneously distributed to the analog path and the digital path to ensure input consistency. 16-bit fixed-point numbers are used to represent weights and intermediate results, balancing precision and resource occupancy (occupying 15% of the DSP Slice resources).

[0085] Among them, the digital convolution algorithm process: Traverse the image with a 3×3 window, and perform multiply-accumulate operations within each window:

[0086] ;

[0087] The zero-padding strategy is adopted to process the image edges to avoid information loss. Through the pipeline architecture of the FPGA, the single-frame digital convolution delay is 1.2 ms (1080P resolution).

[0088] Design and Implementation of Confidence Evaluation Module: The output results of the analog path and the digital path are matched by timestamps to ensure the comparison of convolution results for the same image area. The Euclidean Distance is used to quantify the difference between the two-modal results: ; where S i is the output of the analog path, D i is the output of the digital path, n is the result dimension. According to the requirements of the industrial scenario, the initial threshold is set to 0.1 (normalized distance). The mean μ and standard deviation σ of the distances for 100 consecutive frames are statistically calculated, and the threshold is dynamically adjusted to μ + 2σ. If it is detected that the environmental noise increases, such as the power fluctuation monitored by the current sensor exceeds 5%, the threshold is temporarily reduced to 0.08; if the distance exceeds the threshold, the FPGA control module starts the following process:

[0089] t1. Path Switching: Redistribute the current shard data to another group of memristor sub-arrays or digital logic units for rechecking;

[0090] t2. Result Arbitration: If the rechecked result is consistent with the original result, it is determined as environmental noise interference and the threshold is updated; if not, it is determined as a hardware fault and an alarm is triggered.

[0091] Output high and low level signals through GPIO pins. For example, a 3.3V high level indicates a serious error, driving an external LED or buzzer to prompt the operator.

[0092] The weight adjustment process of the Dynamic Weight Synchronization Network DWS-Net includes: generating a voltage pulse signal through a D / A converter to regulate the initial conductance value of the memristor array; dynamically correcting the conductance distribution according to the real-time image features to match the spatial frequency and gray gradient of the input data; the weight adjustment process realizes the weighted summation of current signals through Kirchhoff's current law to complete the convolution operation.

[0093] It should be further noted that in the specific implementation process, by combining the real-time image features and the dynamic correction of the memristor conductance value, the matching accuracy of the convolution kernel is optimized, thereby improving the accuracy and energy efficiency of image recognition. The specific implementation process is as follows:

[0094] E1. Writing and Verification of Initial Conductance Value: The convolution kernel weights are determined through offline training (such as fine-tuning using the VGG16 model on the industrial defect dataset). Each weight value w ij corresponds to the target conductance value G i , SL j） of the memristor cross-point (BL ij , and the mapping formula is:

[0095] ;

[0096] Where k is the proportional coefficient, which is determined by the conductance range of the memristor, G min It is the minimum conductance value to prevent the conductance from returning to zero.

[0097] The conductance writing process generates a +3V voltage pulse (10ns duration) through a D / A converter (AD9744) and applies it to the target memristor node to increase its conductance value; a -3V voltage pulse (10ns duration) is generated to reduce the conductance value; the pulse amplitude is dynamically adjusted according to the target conductance deviation. For example, when the conductance needs to increase by 20%, the pulse amplitude is increased to +3.5V.

[0098] Conductance value verification and calibration include: reading the SL line current I through a transimpedance amplifier (TIA) immediately after writing SLx , reverse the actual conductance value ;

[0099] If the actual conductance value deviates from the target value by more than 5%, a secondary write or alarm is triggered, that is, a high-level signal is output through the FPGA's GPIO pin. All conductance adjustment records are stored in the FPGA's block RAM for subsequent dynamic correction historical data analysis.

[0100] E2. Real-time image feature analysis and dynamic conductance correction: The FPGA's FFT IP core performs a fast Fourier transform on the input image and calculates the high-frequency energy fraction (the fraction of components above 50 Hz) using the following formula:

[0101] ;

[0102] The Sobel operator is used to calculate the image gradient magnitude, and the percentage of pixels with gradient magnitudes greater than a threshold is counted to represent edge density. The high-frequency energy fraction (8 bits), edge density (8 bits), and average grayscale value (8 bits) are combined into a 24-bit feature vector and transmitted to the conductivity adjustment control unit. In high-frequency-dominated scenarios (i.e., high-frequency energy fraction >60%), the conductance of the edge detection convolution kernel is increased (e.g., by 15%), while the conductance of the smoothing filter kernel is suppressed (by 10%). In low-contrast scenarios (i.e., edge density <20%), the weight of the high-frequency filter kernel is reduced, while the weight of the texture enhancement kernel is increased. Combined with the results of bimodal verification, if the simulated path confidence falls below a threshold, the conductance value is re-adjusted. A linear regression model analyzes historical feature vectors and conductance adjustment records to predict the optimal weight distribution.

[0103] E3. Convolution operation based on Kirchhoff's current law: The grayscale value of the preprocessed image is mapped to a voltage signal V through a D / A converter. BLi , loaded to the bit line BLi of the memristor array.

[0104] Current calculation: According to Kirchhoff's current law, the output current of the source line SLx is:

[0105] ; where, G i is the conductance value after dynamic adjustment, directly corresponding to the convolution kernel weight; the TIA (with a gain of 1000 times) converts the μA-level current signal into a mV-level voltage signal, which is filtered by a high-speed operational amplifier (LMH6521) and then input into the A / D converter; the A / D converter (AD9434) samples the voltage signal with 12-bit precision to generate a digital convolution result, which is transmitted to the FPGA for pooling and fully connected operations.

[0106] The sharding process of the parallel sharded convolution accelerator PSCA includes: dividing the high-resolution image into 4×4 or 8×8 shards, and each shard corresponds to an independent memristor sub-array; the input voltage of each sub-array is generated by the D / A converter based on the shard grayscale value, and the current signal is synchronously output through a transimpedance amplifier; the A / D converter synchronously samples the multi-channel sub-array signals to ensure the timing consistency of the global convolution result.

[0107] It should be further noted that in the specific implementation process, parallel computing of high-resolution images is achieved through sharding processing, specifically including shard segmentation, sub-array voltage generation, transimpedance amplifier synchronous output, and global feature fusion. The implementation process is as follows:

[0108] The dynamic sharding rule setting includes:

[0109] 4×4 shard mode: suitable for high-complexity images, such as micron-level defect detection, dividing the image into a 4×a grid, and the size of each shard is , for example, each shard of a 4K image is 960×540 pixels;

[0110] 8×8 shard mode: suitable for low-complexity scenarios, such as logistics sorting with smooth backgrounds, and the shard size is , for example, each shard of a 4K image is 480×270 pixels;

[0111] The adjacent shards overlap 10% in the horizontal and vertical directions. For example, the 960×540 shard overlaps 96 pixels, and the bicubic interpolation algorithm is used to smooth the edges to avoid information loss;

[0112] The FPGA sharding logic is that the PL end of the FPGA receives the original image data through the AXI-Stream interface, generates a shard index table according to the sharding rules, and stores it in the Block RAM; the shard data is simultaneously transmitted to the D / A converter interfaces of multiple memristor sub-arrays through multiple DMA channels, supporting 64-channel shard parallel processing, adopting a pipeline architecture, and the single-frame shard processing delay <0.4ms, that is, 4K resolution.

[0113] The sub - array input voltage generation and loading include: The 8 - bit grayscale value (0 - 255) of each pixel within a slice is converted into an analog voltage through a formula:

[0114] ;

[0115] where V max is the maximum output voltage of the D / A converter, for example, 1V; A negative voltage (-1V) is generated through a charge pump chip (LM27762) for representing the conductance nodes corresponding to the negative weights of the convolution kernel.

[0116] The D / A converter is configured with a 14 - bit high - precision D / A converter (AD9744), supporting an output range of 0 - 1V and a conversion rate of 100MSPS; The FPGA generates a synchronous clock signal (CLK+ / CLK - ) to control the sampling timing of all sub - array D / A converters, ensuring strict synchronization of slice data loading; The D / A converter of each sub - array is independently controlled through the GPIO pins of the FPGA, supporting dynamic allocation of slices to idle sub - arrays.

[0117] The process of the trans - impedance amplifier synchronously outputting current signals includes: Using a low - noise operational amplifier (OPA657) with a gain set to 1000 times to convert μA - level current signals into mV - level voltage signals. According to the current range, for example, 1μA - 100μA, select R f = 1MΩ, and the output voltage V out = I SL· R f ; A second - order low - pass filter (cut - off frequency 50MHz) is embedded at the output of the TIA to eliminate high - frequency noise; A 32 - channel analog switch (ADG732) is used to switch the SLx signal of the sub - array to the A / D converter in the order of slice indexes. The FPGA attaches a timestamp (counting based on the system clock) to the SLx signal of each slice to ensure data alignment during global fusion.

[0118] For the A / D converter synchronous sampling and global feature fusion: A 12 - bit high - speed A / D converter (AD9434) is used with a sampling rate of 100MSPS, supporting multi - channel synchronous sampling; The PL side of the FPGA generates a synchronous sampling clock (SYNC_CLK) to drive all A / D converters to start sampling simultaneously with a deviation < 1ns; The FPGA reorganizes the convolution results of each slice according to the original spatial position through timestamps to restore the complete feature map; The weight of high - complexity slices is set to 0.7, and the weight of low - complexity slices is 0.3. The formula is:

[0119] ;

[0120] For edge detection tasks, the maximum response value at the corresponding position of each slice is selected. The formula is:

[0121] ;

[0122] Among them, the BLx and SLx of each sub-array are wired through an independent metal layer to avoid signal crosstalk; the power supply of the sub-array is independently controlled by a load switch (TPS22965), and 50% of the sub-arrays are turned off at low load, reducing the power consumption by 35%. The D / A and A / D converters adopt LVDS differential interfaces to enhance the ability to resist common-mode noise. An independent grounding loop is designed for each sub-array to reduce the influence of ground bounce noise.

[0123] The circuit structure of the memristor array computing module includes: a multi-channel D / A converter and a transimpedance amplifier, which are respectively connected to the bit line BLx and the source line SLx of the memristor array;

[0124] The D / A converter regulates the conductance value of the memristor through a voltage pulse signal, and the transimpedance amplifier converts a tiny current signal into a voltage signal;

[0125] After being filtered by a high-speed operational amplifier, the A / D converter converts the analog signal into a digital signal and sends it back to the XC7Z020 FPGA control module.

[0126] It should be further noted that in the specific implementation process, the circuit structure design of the memristor array computing module, including the collaborative work of the multi-channel D / A converter, the memristor array, the transimpedance amplifier and the A / D converter, realizes the analog convolution operation of memory and computing. The implementation process is as follows:

[0127] Selection and configuration of the D / A converter: A 14-bit high-precision D / A converter AD9744 is adopted, which supports 0-1V analog voltage output and has a conversion rate of 100MSPS, meeting the high-speed writing requirements of the memristor array. The bit line (BLx) of each memristor sub-array is connected to an independent D / A converter channel, supporting parallel data loading. For example, a 128×128 memristor array needs to be configured with 128 channels of D / A converters. The D / A converter is controlled by the FPGA to generate ±3V voltage pulses (duration 10ns), which are used to increase (SET) or decrease (RESET) the conductance value of the memristor respectively. The preprocessed image gray value (0-255) is linearly scaled to 0-1V analog voltage and loaded onto the BLx line, representing the input data of the convolution operation.

[0128] The PL end of the FPGA generates a synchronous clock signal (CLK+ / CLK-), driving all D / A converters to sample synchronously to ensure the timing consistency of multi-channel voltage pulses. The digital input of the D / A converter is connected through the Bank34 LVDS differential pair of the FPGA (such as B34_L4_P / N to B34_L10_P / N), supporting high-speed parallel data transmission.

[0129] The physical structure and conductance regulation of the memristor array include: The memristor array is formed by the intersection of BLx (bit line) and SLx (source line). Each intersection integrates a memristor unit, and its conductance value represents the convolution kernel weight. SET / RESET pulses are applied through a D / A converter to adjust the conductance value to a preset range (such as 10 μS - 100 μS). According to the real-time image features, that is, the dynamic weight synchronization network, the conductance value of specific nodes is dynamically corrected.

[0130] When the input voltage is loaded onto the BLx line, the output current of the SLx line is the sum of the products of each BLx voltage and the corresponding conductance value, that is:

[0131] ; This current signal directly implements the convolution operation in the analog domain.

[0132] The current-voltage conversion of the transimpedance amplifier (TIA) includes: Using the low-noise transimpedance amplifier OPA657 with a gain-bandwidth product of 1.6 GHz and an input bias current of 1 pA, which is suitable for amplifying μA-level current signals. Set the feedback resistor R f = 1 MΩ according to the current range (1 μA - 100 μA), and the output voltage V out = I SLx· R f ; Connect a 0.1 μF ceramic capacitor to the power supply pin of OPA657 to suppress high-frequency noise; among them, the SLx signal line uses differential routing to reduce electromagnetic interference.

[0133] Use a 32-channel analog switch ADG732 to switch the SLx signal to the A / D converter according to the shard index, and the switching delay < 10 ns. The FPGA adds a timestamp to each SLx signal to ensure data alignment during global feature fusion.

[0134] A / D conversion and high-speed op-amp filtering include: Configure the A / D converter, use a 12-bit high-speed A / D converter AD9434 with a sampling rate of 100 MSPS, support differential input, and a dynamic range of 70 dB. Connect a high-speed op-amp LMH6521 at the front end of the A / D converter and configure it as a second-order low-pass filter (cutoff frequency 50 MHz) to eliminate high-frequency noise. Adjust the 0 - 1 V signal output by the TIA to the input range (0 - 2 V) of AD9434 through a resistor voltage divider network. The differential output (D0+ / - to D11+ / -) of AD9434 is connected to the PL end through the Bank35 LVDS pins (B35_L2_P / N to B35_L14_P / N) of the FPGA. The FPGA generates a synchronous sampling signal (SYNC) to ensure strict synchronization between the A / D converter and the D / A converter, and the sampling deviation < 1 ns.

[0135] Regarding this, the system collaborative working process includes:

[0136] Data Loading and Convolution Operation:

[0137] Step 1: The FPGA distributes the preprocessed image shard data to multiple D / A converters to generate corresponding BLx voltage signals;

[0138] Step 2: The memristor array completes analog convolution calculations according to the conductance values, and the SLx output current signal is converted into a voltage signal by the TIA;

[0139] Step 3: The A / D converter samples the filtered analog signal into a 12-bit digital result and transmits it back to the FPGA for global fusion and post-processing.

[0140] Closed-loop Verification and Calibration: Regularly read the SLx current to reverse the actual conductance value, compare it with the target value, and trigger a secondary write when the deviation exceeds 5%; through the noise power spectral density analysis of the AD9434, dynamically adjust the filter parameters to ensure signal integrity.

[0141] The process by which the IoT module receives instructions through the CAN bus includes: The XC7Z020 FPGA control module generates action instructions based on the fully connected results of the convolutional neural network; the instructions are transmitted to the robotic arm drive circuit through the CAN_H and CAN_L signals to perform sorting, positioning, or obstacle avoidance operations.

[0142] It should be further noted that in the specific implementation process, the complete process by which the IoT module receives instructions through the CAN bus includes FPGA generating action instructions, CAN protocol encapsulation, data transmission, and the robotic arm executing operations. Its implementation process covers hardware circuit design, communication protocol implementation, and closed-loop control mechanism, as follows:

[0143] The process of the FPGA generating action instructions: The PL side of the FPGA integrates the pre-trained fully connected layer weight matrix (stored in the Block RAM), completes matrix multiplication through the parallel multiplier-adder (DSP48E1 Slice), and outputs classification probabilities or coordinate information. It includes: For the defect detection task, the output is the defect category, such as "loose solder", "short circuit", and its coordinates (x, y) in the image. The classification result and coordinate information are encoded into a 32-bit data packet, and the format is: data packet = [category encoding (4 bits) | coordinate X (12 bits) | coordinate Y (12 bits) | check bit (4 bits)]; among them, the check mechanism uses the CRC-4 check algorithm to ensure the integrity of data transmission.

[0144] For instruction priority scheduling, the PS side of the FPGA maintains an instruction queue and dynamically adjusts the priority according to the task urgency. It includes: High priority: The robotic arm obstacle avoidance instruction (response delay < 2 ms). Low priority: The sorting or positioning instruction (response delay < 5 ms). Interrupt handling: If an emergency event (such as a collision risk) is detected, a high-priority instruction is immediately triggered and inserted into the head of the queue.

[0145] For the implementation of the CAN bus communication protocol, in the hardware circuit design, the CAN controller is configured as follows: The PS side of the XC7Z020 FPGA integrates a CAN controller IP core (such as Xilinx AXI CAN), which supports the CAN 2.0B protocol and has a communication rate of 1 Mbps; for the CAN transceiver in the physical layer interface: The TI SN65HVD230 chip is used to convert the TTL signal of the FPGA into differential signals (CAN_H / CAN_L). A 120Ω termination resistor is connected at both ends of the CAN bus to suppress signal reflection. The CAN controller of the FPGA is connected to the robotic arm drive circuit through the MIO pins (B1_MIO41: CAN_H, B1_MIO42: CAN_L). Among them, a frame of data is sent every 5 ms to avoid bus congestion; if the ACK response of the robotic arm is not received, the FPGA automatically resends the instruction after 1 ms, with a maximum of 3 retries.

[0146] Robotic arm drive and action execution: After the robotic arm drive circuit analyzes the CAN instruction, it generates a PWM signal through a microcontroller, such as STM32F407, to control the servo motor to rotate to the target angle. The coordinates (x, y) are converted into joint angles (θ1, θ2) through the inverse kinematics model. An optical encoder (such as OMRON E6B2-CWZ6C) is used to feedback the actual angle to achieve closed-loop control (accuracy ±0.1°). Multiple servo motors are synchronously controlled through the EtherCAT bus to ensure the smooth movement of the end effector of the robotic arm.

[0147] Perform sorting and positioning operations: According to the weight of the target object (matched through a preset parameter library), adjust the air pump pressure (0 - 0.6 MPa). And a high-friction coefficient silica gel is covered on the surface of the gripper, and the pressure sensor Honeywell FSS1500NGT monitors the clamping force in real time. When the limit is exceeded, an alarm is triggered; a micro camera OV9281 is integrated at the end of the robotic arm, and the position is secondarily calibrated through the real-time image transmitted back by the FPGA, and the positioning error < 0.05 mm.

[0148] For obstacle avoidance path planning, the distance to surrounding obstacles is measured in real time by a ToF sensor (such as VL53L1X) (detection range 0 - 4m, accuracy ±1cm). The path planning algorithm pre-sets a grid map in the FPGA through the A* algorithm and generates an optimal path based on the positions of the obstacles. If a sudden obstacle is detected, the path is re-planned through the RRT* (rapidly-exploring random tree) algorithm, and the response time is <10ms. Mechanical limit switches are set at the joints of the robotic arm to forcibly limit the movement range and prevent hardware damage.

[0149] Closed-loop feedback and exception handling are performed, that is, the drive circuit of the robotic arm collects the following parameters in real time and transmits them back to the FPGA through the CAN bus: joint angle, end position, clamping force, power supply voltage, temperature.

[0150] The data frame format is the transmission frame = [status code (4 bits) | joint angle (24 bits) | clamping force (12 bits) | CRC-8]; if it is detected that the clamping force exceeds the limit, that is, >50N, or the temperature is too high, that is, >80°C, the robotic arm immediately stops moving and sends an emergency alarm frame (ID: 0x18FF0001); after receiving the alarm, the FPGA triggers the system self-check process, resets the faulty module and restores the initial state.

[0151] The dynamic resource allocation mechanism optimizes the platform energy consumption through a load balancing algorithm, including: when it is detected that the complexity of consecutive multi-frame images is lower than the preset threshold, the power supply of some memristor sub-arrays is turned off to reduce power consumption; in high-load scenarios, redundant sub-arrays of the parallel sharding convolutional accelerator PSCA are enabled for hyper-thread acceleration.

[0152] It should be further noted that in the specific implementation process, the enabled state of the memristor sub-arrays is dynamically adjusted through a load balancing algorithm to optimize the system energy consumption, specifically including turning off some sub-arrays at low load to reduce power consumption, and enabling redundant sub-arrays for hyper-thread acceleration at high load to increase throughput. Its implementation process covers load monitoring, resource scheduling, and hardware control, as follows:

[0153] A lightweight algorithm is embedded in the PS side (Processing System) of the FPGA to calculate the key features of the input image in real time: calculate the high-frequency energy ratio through the fast Fourier transform (FFT), and the energy of the components above 50Hz accounts for the percentage of the total energy.

[0154] The Sobel operator is used to count the proportion of pixels whose gradient magnitude exceeds the threshold, and the entropy value of the image grayscale histogram is calculated to characterize the information complexity. The above features are normalized to a complexity score (Score) between 0 and 100, and the formula is: Score = 0.4 × high-frequency energy ratio + 0.3 × edge density + 0.3 × grayscale entropy. Among them, the load status determination includes: if the average value of Score for 10 consecutive frames of images < 40, it is determined as low load; if the Score of a single frame > 80 or the average value of Score for 3 consecutive frames > 70, it is determined as high load.

[0155] During the energy consumption optimization in the low-load mode, for the sub-array shutdown strategy, the sharding mapping relationship is as follows: according to the sharding processing mode, such as 4×4 or 8×8 sharding, each shard is bound to a specific sub-array group. Among them, the dynamic shutdown process includes: the FPGA sends a control signal to the power management chip TI TPS22965 through the GPIO pin to turn off the power supply of the idle sub-array, and only the core sub-array is retained, such as 1 / 4 of the total array working, and the rest of the sub-arrays enter the sleep state.

[0156] In the power management circuit, the power supply of each sub-array is controlled by an independent load switch, which supports start and stop in the nanosecond level. The static power consumption of the sleeping sub-array is reduced from 50mW to 5μW, a decrease of 99%.

[0157] In task reallocation and calculation compensation, sharding merging is carried out, including: merging the shards originally assigned to the shutdown sub-array to the adjacent active sub-array, maintaining the processing ability by increasing the shard size (such as merging from 8×8 to 4×4), and the input voltage of the active sub-array is increased proportionally (such as 1.2 times) to compensate for the increased calculation volume after sharding merging. The results are weighted and integrated through the global fusion algorithm to avoid accuracy loss.

[0158] For the hyper-threading acceleration in the high-load mode, during the redundant sub-array enabling mechanism, the system reserves 20% of the memristor sub-arrays as redundant resources, which are in the sleep state under normal conditions.

[0159] The dynamic enabling process includes: after the FPGA detects high load, it wakes up the redundant sub-array through the power management chip; the redundant sub-array accesses the computing network and processes the same shard data in parallel with the original shard-bound sub-array (hyper-threading mode).

[0160] Hyper-thread acceleration implementation includes: copying the same sharded data to multiple sub-arrays, parallelly extracting multi-dimensional features through different convolutional kernel weights (such as edge detection kernel and texture enhancement kernel), selecting the maximum value of the outputs of each sub-array as the final feature to enhance the edge detection effect, dynamically allocating weights according to the historical accuracy of the sub-arrays to improve the reliability of the fusion result, coordinating the computing timing of redundant sub-arrays through the global clock signal (such as 200MHz) of the FPGA to avoid data conflicts, preferentially allocating highly complex shards to redundant sub-arrays to ensure the processing speed of critical tasks.

[0161] During the implementation of the load balancing algorithm and energy consumption control, design the algorithm logic: input the real-time complexity score (Score), historical energy consumption data, and sub-array status; output the start / stop instructions of the sub-arrays, voltage regulation parameters, and task allocation strategies.

[0162] The core steps include: first, calculate the current frame Score and update the moving window mean (window size = 10 frames); if the mean < 40 and lasts for 5 frames, trigger the low-load mode and turn off 50% of the sub-arrays; if the single-frame Score > 80, trigger the high-load mode and enable all redundant sub-arrays, dynamically adjust the voltage of the sub-arrays according to the real-time power consumption (collected by the current sensor MAX44284) to ensure that the total power consumption ≤ 5W.

[0163] If the real-time power consumption exceeds the threshold (such as 5.5W), automatically reduce the voltage of non-critical sub-arrays (such as from 1V to 0.8V); if the power consumption is lower than the target (such as 4W), gradually increase the voltage in exchange for higher computing accuracy, analyze the relationship between historical load and energy consumption through a machine learning model (such as linear regression) to predict the optimal resource allocation strategy.

[0164] This data analysis platform significantly improves the speed, energy efficiency, and reliability of image processing through the deep integration of the memristor-based memory and computing architecture and dynamic optimization technology. First, the memristor array implements parallel convolutional operations in an analog manner, eliminating the data transfer bottleneck of the traditional von Neumann architecture, improving the processing speed compared to the traditional GPU solution, and reducing the system energy consumption at the same time. The dynamic weight synchronization network and parallel sharded convolutional accelerator further optimize the computing efficiency, adjust the weight allocation and sharded processing strategy through real-time feature feedback, improving the recognition accuracy and throughput in complex industrial scenarios, and meeting the real-time processing requirements of high-resolution images.

[0165] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0166] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An image processing data analysis platform based on a memristor storage and computing integrated architecture, characterized in that: include: The XC7Z020 FPGA control module is connected to the OV5640 camera image acquisition module, the memristor array computing module, and the Internet of Things module; The OV5640 camera image acquisition module is used to collect raw image data and transmit it to the XC7Z020 FPGA control module; The memristor array computing module includes a D / A converter, a memristor array, a transimpedance amplifier and an A / D converter, and a dynamic weight synchronization network DWS-Net, which is used to receive image data processed by the XC7Z020 FPGA control module, complete convolution operations through parallel analog computing, and return the results to the XC7Z020 FPGA control module; The IoT module receives instructions from the XC7Z020 FPGA control module via the CAN bus and executes predefined actions; The platform also includes a dynamic resource allocation mechanism that integrates a real-time task scheduling unit through the XC7Z020 FPGA control module to dynamically allocate computing tasks to FPGA logic units or the memristor array computing module based on the complexity of the input image; The real-time task scheduling unit generates task priorities by analyzing the spatial frequency, edge density, and grayscale distribution of the image, and allocates convolution operations in high-complexity areas to the memristor array computing module for parallel analog computing, while retaining low-complexity processing tasks for execution in the FPGA logic unit; The weights of the memristor array are pre-written in the form of conductance and loaded to the array nodes via voltage signals, thus achieving convolution operations that integrate storage and computation. The XC7Z020 FPGA control module deploys a dual-modal verification mechanism, DMVM. It constructs a memristor array simulation convolution operation path for the first modality and an FPGA digital logic verification path for the second modality. The confidence assessment module compares the calculation results of the two modalities. If the difference exceeds a preset threshold, a secondary calculation or an alarm is triggered. The weight adjustment process of the dynamic weight synchronization network DWS-Net includes: generating a voltage pulse signal through the D / A converter to regulate the initial conductance value of the memristor array; dynamically correcting the conductance distribution according to real-time image features to match the spatial frequency and grayscale gradient of the input data; and completing the convolution operation by implementing the weighted summation of the current signal through Kirchhoff's current law.

2. The image processing data analysis platform based on the memristor storage and computing integrated architecture according to claim 1, characterized in that: The XC7Z020 FPGA control module is used to segment, compress and grayscale the original image to generate pre-processed data; The pre-processed data is transmitted to the memristor array computing module, and the robotic arm is controlled to perform actions.

3. The image processing data analysis platform based on the memristor storage and computing integrated architecture according to claim 2, characterized in that: The XC7Z020 FPGA control module is embedded with a lightweight feature extraction unit, which analyzes the spatial frequency and texture distribution of the input image in real time to generate a feature vector. Based on the feature vector, the conductance value of different nodes in the memristor array is adjusted through the D / A converter.

4. The image processing data analysis platform based on the memristor storage and computing integrated architecture according to claim 3, characterized in that: The memristor array computing module integrates a parallel sliced convolution accelerator (PSCA), namely: the XC7Z020 FPGA control module divides the input image into N×N independent sub-regions, which are regarded as slices. Each slice is assigned to an independent subarray of the memristor array; each subarray completes local convolution operations through parallel analog computing and synchronously outputs current signals through the multiplexing technology of transimpedance amplifiers; the A / D converter converts the analog signals of multiple subarrays into digital signals, and the XC7Z020 FPGA control module performs global feature fusion.

5. The image processing data analysis platform based on the memristor storage and computing integrated architecture according to claim 4, characterized in that: The slice processing process of the parallel slice convolution accelerator (PSCA) includes: dividing the high-resolution image into 4×4 or 8×8 slices, each slice corresponding to an independent memristor subarray; the input voltage of each subarray is generated by the D / A converter based on the slice grayscale value, and the current signal is synchronously output through the transimpedance amplifier; the A / D converter synchronously samples the multi-channel subarray signals.

6. The image processing data analysis platform based on the memristor storage and computing integrated architecture according to claim 5, characterized in that: The circuit structure of the memristor array computing module includes: a multi-channel D / A converter and a transimpedance amplifier, which are respectively connected to the bit line BLx and the source line SLx of the memristor array; The D / A converter controls the conductance of the memristor through a voltage pulse signal, and the transimpedance amplifier converts a tiny current signal into a voltage signal; The A / D converter converts the analog signal into a digital signal after filtering by a high-speed operational amplifier and transmits it back to the XC7Z020FPGA control module.

7. The image processing data analysis platform based on the memristor storage and computing integrated architecture according to claim 6, characterized in that: The process of the IoT module receiving instructions through the CAN bus includes: the XC7Z020 FPGA control module generates action instructions based on the fully connected results of the convolutional neural network; the instructions are transmitted to the robotic arm drive circuit through the CAN_H and CAN_L signals to perform sorting, positioning or obstacle avoidance operations.

8. The image processing data analysis platform based on the memristor storage and computing integrated architecture according to claim 7, characterized in that: The dynamic resource allocation mechanism optimizes platform energy consumption through a load balancing algorithm, including: when it is detected that the complexity of multiple consecutive frames of images is lower than a preset threshold, the power supply of some memristor subarrays is turned off to reduce power consumption; in high-load scenarios, the redundant subarrays of the parallel sliced convolution accelerator (PSCA) are enabled for hyperthreading acceleration.

Images