Motion information extraction and fast optical flow calculation system based on memristor

Abstract

The invention relates to the technical field of computer vision and neuromorphic calculation, in particular to a memristor-based motion information extraction and fast optical flow calculation system, which comprises a front-end imaging module, a rear-end imaging module, a memory module and a signal processing module, and is characterized in that the front-end imaging module is used for acquiring a visual scene and converting the visual scene into an analog voltage signal; the voltage conversion module is used for extracting a light intensity change condition based on the analog voltage signal; the neuromorphic module is used for coding time dynamic information of a visual scene according to the light intensity change condition; and the signal processing module is used for extracting a time motion clue based on the time dynamic information and carrying out optical flow calculation on a corresponding region of the time motion clue. According to the invention, efficient motion analysis can be realized, and the optical flow calculation speed is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and neuromorphic computing technology, and in particular to a memristor-based motion information extraction and fast optical flow calculation system, which is suitable for dynamic scenarios with strict real-time requirements, such as drone obstacle avoidance, autonomous driving, and robot grasping. Background Art

[0002] Traditional motion perception relies on visual sensors (such as cameras) capturing consecutive frames of images and inferring the motion of objects in the scene by analyzing changes in pixel intensity. Inspired by biological vision, optical flow technology is widely used to calculate spatial motion vectors. Its core goal is to estimate the speed and direction of an object by using the pixel displacement between adjacent frames. As one of the core algorithms in computer vision, optical flow provides a key motion representation for dynamic scene understanding by quantifying the motion vector field of pixels in an image sequence. Its technical principles are based on the assumptions of brightness constancy, spatiotemporal continuity, and minimal motion, and it solves for the two-dimensional displacement vector of each pixel by constructing an energy function. Currently, mainstream optical flow algorithms can be divided into two categories: traditional methods, which use spatial gradient analysis (such as the Farneback algorithm) to calculate the displacement field from the difference in pixel intensity between adjacent frames; and deep learning methods, which use end-to-end neural network-based models (such as RAFT and GMFlow) to extract features and predict the optical flow field. While these methods achieve near-human-level accuracy on benchmark datasets (such as KITTI and Sintel), their high computational complexity and reliance on high-performance hardware (such as GPUs) lead to significant latency in practical deployment.

[0003] Current optical flow computation methods face three core challenges in engineering implementation, severely hindering their in-depth application in real-time systems: a) High latency: As deep learning models become more complex (e.g., RAFT requires 18 optimization iterations), computational time increases exponentially, making it difficult to meet the stringent real-time requirements of scenarios like autonomous driving (typically less than 10ms). This latency poses a serious safety hazard in dynamic environments: when a vehicle is traveling at 80 km / h, a delay of just 0.6 seconds can increase braking distance by an additional 27 meters, equivalent to the blind spot of five standard vehicles. b) Limitations of pure spatial analysis: Current algorithms primarily rely on spatial gradient analysis, which suffers from two fundamental flaws. First, traditional methods, based on the assumption of constant brightness, often ignore dynamic temporal evolution (e.g., nonlinear characteristics of motion trajectories). This makes it difficult to distinguish between background noise (e.g., swaying leaves) and real moving objects (e.g., pedestrians crossing the road) in scenes with sudden changes in illumination or motion blur. Second, existing methods lack effective simulation of the "attention focusing" mechanism of biological vision, forcing them to perform uniform computations across the entire image, resulting in significant computational waste in non-critical areas. c) Hardware Adaptability Deficiencies: The architectural design of traditional computing platforms severely restricts energy efficiency improvements. Systems based on the von Neumann architecture suffer from the "memory wall" problem. Due to the physical separation of storage and computing units, a significant amount of energy is wasted in data transfer. Furthermore, fixed circuit designs relying on CMOS processes struggle with dynamic reconfiguration of computing logic, making adaptive perception optimization impossible in complex environments. Therefore, this paper proposes a memristor-based motion information extraction and fast optical flow calculation system. Summary of the Invention

[0004] The purpose of the present invention is to provide a memristor-based motion information extraction and fast optical flow calculation system, based on the motion information encoding mechanism of neuromorphic synaptic arrays, to achieve efficient motion analysis by directly embedding spatiotemporal information at the hardware level.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] Memristor-based motion information extraction and fast optical flow calculation system, including:

[0007] Front-end imaging module, used to capture visual scenes and convert them into analog voltage signals;

[0008] A voltage conversion module, configured to extract light intensity changes based on the analog voltage signal;

[0009] a neuromorphic module for encoding temporal dynamic information of a visual scene through the light intensity variation;

[0010] The signal processing module is used to extract temporal motion clues based on the temporal dynamic information and perform optical flow calculation on the corresponding area of ​​the temporal motion clues.

[0011] Optionally, the front-end imaging module adopts a front-end imaging array. After the front-end imaging array collects the visual scene and converts it into an analog voltage signal, one path is converted into a digital image through analog-to-digital conversion, and the other path is input to the voltage conversion module.

[0012] Optionally, the voltage conversion module includes:

[0013] a differential processing unit, configured to extract an absolute value of light intensity variation based on the analog voltage signal;

[0014] The amplitude conversion unit is used to extract the relative change amplitude of the light intensity based on the absolute value of the light intensity change, and generate a corresponding pulse according to the relative change amplitude of the light intensity.

[0015] Optionally, the differential processing unit extracting the absolute value of the light intensity change based on the analog voltage signal includes:

[0016]

[0017] Among them, V i，j (t) is the absolute value of the light intensity change at time t, a is the proportional coefficient, I i,j is the input light intensity, i and j represent the photosensitive element at position (i, j) in the front-end imaging array, I i，j (t) is the input light intensity at time t, t is the sampling time, and Δt is the sampling interval.

[0018] Optionally, the amplitude conversion unit generates corresponding pulses from the relative change amplitude of the light intensity, including:

[0019]

[0020] in, is the modulated pulse corresponding to the relative change in light intensity, V th is the reference voltage for comparison, bia1, bia2, plus1, plus2 are built-in parameters in the reconfigurable operational amplifier, is the relative change in light intensity, and i and j represent the photosensitive element at position (i, j) in the front-end imaging array.

[0021] Optionally, the neuromorphic module adopts a neuromorphic synaptic array, which includes a resistive random access memory, a two-terminal memristor, and a three-terminal synaptic transistor. The neuromorphic synaptic array is driven by the corresponding pulse generated by the voltage conversion module to enter a corresponding conductance state to obtain state distribution information.

[0022] Optionally, the signal processing module extracting temporal motion cues based on the temporal dynamic information includes:

[0023] Converting the state distribution of the neuromorphic synaptic array into a grayscale image via logarithmic mapping;

[0024] The motion area is divided in the grayscale image by comparing with a preset threshold, and a binary mask representing the temporal motion clue is generated by combining 8-neighborhood connectivity analysis and morphological dilation.

[0025] Optionally, the signal processing module performing optical flow calculation on the area corresponding to the temporal motion clue includes:

[0026] The binary mask marked area of ​​temporal motion clues is used as the region of interest;

[0027] Select the corresponding optical flow calculation method based on the application scenario, perform optical flow calculation using the region of interest, and obtain two-dimensional optical flow information;

[0028] The two-dimensional optical flow information is combined with the binary mask to form a three-dimensional neuromorphic optical flow.

[0029] The beneficial effects of the present invention are:

[0030] (1) Ultra-low latency, collaborative optimization of hardware-level time encoding and ROI screening:

[0031] By leveraging the temporal motion cues output by the synaptic array, the system only needs to perform optical flow calculations on 10%-30% of the image area, eliminating the need for full image processing and directly reducing algorithm complexity. This reduces overall system processing time from 0.6 seconds using traditional methods to under 150ms, a fourfold increase (400% acceleration), meeting the real-time requirements of scenarios such as autonomous driving and drones.

[0032] (2) Accuracy improvement, noise suppression and prior enhancement under spatiotemporal consistency constraints:

[0033] The synaptic array extracts the temporal gradient of light intensity changes through differential circuits, and combines it with the spatial gradient to form a spatiotemporal consistency constraint, which can distinguish real motion from background noise. In subsequent tasks, the temporal motion cues can provide ROI boundary constraints for the neural network, solving the generalization problem of traditional methods in untrained scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1This is a schematic diagram of the structure of a memristor-based motion information extraction and fast optical flow calculation system according to an embodiment of the present invention;

[0036] Figure 2 This is a flow chart of temporal motion clue extraction according to an embodiment of the present invention;

[0037] Figure 3 This is a flowchart of optical flow calculation according to an embodiment of the present invention;

[0038] Figure 4 This is a diagram showing the application results of the memristor-based motion information extraction and fast optical flow calculation system in multiple scenarios using multiple optical flow algorithms according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] This embodiment provides a memristor-based motion information extraction and fast optical flow calculation system, including:

[0042] Front-end imaging module, used to capture visual scenes and convert them into analog voltage signals;

[0043] A voltage conversion module, configured to extract light intensity changes based on the analog voltage signal;

[0044] a neuromorphic module for encoding temporal dynamic information of a visual scene through the light intensity variation;

[0045] The signal processing module is used to extract temporal motion clues based on the temporal dynamic information and perform optical flow calculation on the corresponding area of ​​the temporal motion clues.

[0046] Specifically, such as Figure 1 As shown in Figure 1, the system's core architecture includes a front-end imaging array, a neuromorphic synaptic array, a voltage conversion circuit, and a signal processing algorithm module. Through the collaborative design of hardware and algorithms, the system enables rapid encoding and processing of spatiotemporal motion information in visual scenes, significantly improving the efficiency of optical flow calculations.

[0047] Furthermore, the front-end imaging module adopts a front-end imaging array. After the front-end imaging array collects the visual scene and converts it into an analog voltage signal, one path is converted into a digital image through analog-to-digital conversion, and the other path is input to the voltage conversion module.

[0048] Specifically, in this embodiment, the front-end imaging array uses a commercial camera or custom sensor to convert the external visual scene into an analog voltage signal. The imaging array output is divided into two paths: one path generates a digital image through analog-to-digital conversion; the other path is input to a voltage conversion circuit to drive the neuromorphic synapse array.

[0049] Furthermore, the voltage conversion module includes:

[0050] a differential processing unit, configured to extract an absolute value of light intensity variation based on the analog voltage signal;

[0051] The amplitude conversion unit is used to extract the relative change amplitude of the light intensity based on the absolute value of the light intensity change, and generate a corresponding pulse according to the relative change amplitude of the light intensity.

[0052] Specifically, in this embodiment, the differential processing unit is composed of a high-pass filter and an operational amplifier, and the absolute value of the light intensity change is extracted, and the expression is:

[0053]

[0054] Where I is the input light intensity, Δt is the sampling interval, a is the proportional coefficient, and i and j represent the photosensitive element at position (i, j) in the front-end imaging array.

[0055] The absolute value circuit in the amplitude conversion unit first converts the above differential voltage, that is, the absolute value of the light intensity change V i,j Perform absolute value calculation to extract the relative change in light intensity, as shown below:

[0056]

[0057] The reconfigurable operational amplifier in the amplitude conversion module further generates the modulation pulse.

[0058]

[0059] Among them, V th is the reference voltage for comparison, bia1, bia2, plus1, plus2 are built-in parameters in the reconfigurable operational amplifier. In this circuit, when the threshold V thNegative pulses are generated in areas of significant motion, driving the neuromorphic synaptic device into a high-conductance state; background noise generates positive pulses to suppress interference. The pulse polarity here is not fixed and needs to be considered in conjunction with the electrical characteristics of the neuromorphic synaptic device. Its core computational essence is to distinguish between changes in light intensity caused by external motion information. Sharp changes correspond to pulses that modulate the neuromorphic synaptic device to a low-impedance state, while slow changes correspond to pulses that modulate the neuromorphic synaptic device to a high-impedance state.

[0060] Furthermore, the neuromorphic module adopts a neuromorphic synaptic array, which includes a resistive random access memory, a two-terminal memristor, and a three-terminal synaptic transistor. The neuromorphic synaptic array is driven into a corresponding conductance state by a corresponding pulse generated by the voltage conversion module to obtain state distribution information.

[0061] Specifically, the neuromorphic synaptic array in this embodiment can utilize a variety of neuromorphic devices, including resistive random access memory (RRAM), two-terminal memristors, and three-terminal synaptic transistors. During actuation, the conductance state of the neuromorphic device changes, directly encoding the temporal dynamics of the visual scene.

[0062] Furthermore, the signal processing module extracts temporal motion clues based on the temporal dynamic information, including:

[0063] Converting the state distribution of the neuromorphic synaptic array into a grayscale image via logarithmic mapping;

[0064] The motion area is divided in the grayscale image by comparing with a preset threshold, and a binary mask representing the temporal motion clue is generated by combining 8-neighborhood connectivity analysis and morphological dilation.

[0065] Specifically, such as Figure 2 As shown, in this embodiment, the state distribution of the neuromorphic synaptic array is converted into a 0-255 grayscale image through logarithmic mapping, and the formula is:

[0066]

[0067] Here, s is the result of the conversion, state is the collected state information (which can be the resistance state of the synaptic device or the current flowing through it), and a and b are parameters that need to be determined based on the actual device conditions. Ultimately, the device state distribution is converted into a 0-255 grayscale image. This mapping optimizes the dynamic range by enhancing the contrast in low-current regions and suppressing saturation at high values. The motion regions are then demarcated by comparing them with a threshold. Combining 8-neighborhood connectivity analysis with morphological dilation, a binary mask representing temporal motion cues is generated. This temporal motion cue reduces the complexity of subsequent optical flow calculations and task execution algorithms, thereby achieving efficient human-like motion information processing.

[0068] Furthermore, the signal processing module performs optical flow calculation on the area corresponding to the temporal motion clue, including:

[0069] The binary mask marked area of ​​temporal motion clues is used as the region of interest;

[0070] Select the corresponding optical flow calculation method based on the application scenario, perform optical flow calculation using the region of interest, and obtain two-dimensional optical flow information;

[0071] The two-dimensional optical flow information is combined with the binary mask to form a three-dimensional neuromorphic optical flow.

[0072] Specifically, in the optical flow calculation process, compared with the traditional optical flow calculation based on spatial information only, this implementation regards the area marked by the temporal motion clue binary mask as the region of interest (ROI), and performs optical flow calculation on the ROI area, such as Figure 3 As shown, this implementation is highly compatible and can be combined with different optical flow calculation methods. For example, traditional methods such as Farneback only perform quadratic polynomial fitting on pixels within the ROI, while neural network methods such as GMFlow perform global matching within the ROI through the attention mechanism, with a complexity ranging from O(N 2 ) is reduced to O(M 2 ), where M is much smaller than N. The RAFT algorithm iteratively updates the optical flow field through local correlation volumes. These three optical flow calculation methods are suitable for different application scenarios. Farneback is suitable for lightweight scenarios, while RAFT is more suitable for analyzing complex scenes. The high compatibility of this implementation ensures that the most appropriate algorithm can be selected according to the scenario in subsequent use.

[0073] After the two-dimensional motion gradient information in space is solved by the optical flow algorithm, the two-dimensional optical flow information can be combined with the binary mask based on the state of the neuromorphic synaptic device to form a three-dimensional neuromorphic optical flow.

[0074] System application and verification:

[0075] The system proposed in this embodiment integrates temporal and spatial motion cues (calculated spatial motion gradients) to support the basic tasks of self-driving cars, drones, and robots to autonomously and intelligently perceive, understand, and interact with their surroundings. During task execution, these motion cues are first decomposed. The temporal motion cues then selectively filter the spatial cues so that they only focus on areas with potential motion information, rather than processing the entire scene. This selective focus significantly speeds up the execution of subsequent tasks. The filtered spatial cues are then combined with the visual input to execute task-specific algorithms. Figure 4 As shown, supported tasks include motion prediction, object segmentation, and object tracking.

[0076] The processing results show that the entire process (detection + calculation + task) takes an average of 150ms, which is four times faster than the traditional method (600ms); in terms of accuracy, this system can maintain the accuracy of each task or even improve it. In certain scenarios, such as vehicle operation, small drone operation and grasping, based on the additional environmental knowledge embedded in the time cues, the accuracy rates are improved by 213.5%, 157.4% and 740.9% respectively.

[0077] This system, based on the motion information encoding mechanism of neuromorphic synaptic arrays, achieves efficient motion analysis by directly embedding spatiotemporal information at the hardware level. While acquiring the spatial motion vectors of the visual scene, it leverages the non-volatile storage characteristics of the neuromorphic synaptic array to capture dynamic features in the temporal dimension in real time, locking onto key motion areas within 1-2 milliseconds. This represents a 400% speed increase compared to traditional purely spatial optical flow algorithms. This system eliminates the need for iterative optimization processes associated with traditional deep learning models. Operators simply deploy a vision module integrated with this system to enable robots to simultaneously complete motion perception and decision-making in natural activity scenarios (such as high-speed obstacle avoidance and dynamic grasping). Its spatiotemporal consistency effectively addresses safety concerns such as the 27-meter increase in braking distance caused by delays in existing algorithms. The system's embedded processing capabilities and hardware energy efficiency advantages make it particularly suitable for promotion and application on mobile platforms with stringent real-time requirements, such as drones and service robots.

[0078] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A memristor-based motion information extraction and fast optical flow calculation system, characterized by: include: Front-end imaging module, used to capture visual scenes and convert them into analog voltage signals; A voltage conversion module, configured to extract light intensity changes based on the analog voltage signal; a neuromorphic module for encoding temporal dynamic information of a visual scene through the light intensity variation; The signal processing module is used to extract temporal motion clues based on the temporal dynamic information and perform optical flow calculation on the corresponding area of ​​the temporal motion clues.

2. The memristor-based motion information extraction and fast optical flow calculation system according to claim 1, characterized in that: The front-end imaging module adopts a front-end imaging array. After the front-end imaging array collects the visual scene and converts it into an analog voltage signal, one path is converted into a digital image through analog-to-digital conversion, and the other path is input to the voltage conversion module.

3. The memristor-based motion information extraction and fast optical flow calculation system according to claim 1, characterized in that: The voltage conversion module includes: a differential processing unit, configured to extract an absolute value of light intensity variation based on the analog voltage signal; The amplitude conversion unit is used to extract the relative change amplitude of the light intensity based on the absolute value of the light intensity change, and generate a corresponding pulse according to the relative change amplitude of the light intensity.

4. The memristor-based motion information extraction and fast optical flow calculation system according to claim 3, characterized in that: The differential processing unit extracting the absolute value of the light intensity change based on the analog voltage signal includes: Among them, V i,j (t) is the absolute value of the light intensity change at time t, a is the proportional coefficient, I i,j is the input light intensity, i and j represent the photosensitive element at position (i, j) in the front-end imaging array, I i,j (t) is the input light intensity at time t, t is the sampling time, and Δt is the sampling interval.

5. The memristor-based motion information extraction and fast optical flow calculation system according to claim 3, characterized in that: The amplitude conversion unit generates corresponding pulses from the relative change amplitude of the light intensity, comprising: in, is the modulated pulse corresponding to the relative change in light intensity, V th is the reference voltage for comparison, bia1, bia2, plus1, plus2 are built-in parameters in the reconfigurable operational amplifier, is the relative change in light intensity, and i and j represent the photosensitive element at position (i, j) in the front-end imaging array.

6. The memristor-based motion information extraction and fast optical flow calculation system according to claim 1, characterized in that: The neuromorphic module adopts a neuromorphic synaptic array, which includes a resistive random access memory, a two-terminal memristor, and a three-terminal synaptic transistor. The neuromorphic synaptic array is driven into a corresponding conductance state by a corresponding pulse generated by the voltage conversion module to obtain state distribution information.

7. The memristor-based motion information extraction and fast optical flow calculation system according to claim 6, characterized in that: The signal processing module extracts temporal motion clues based on the temporal dynamic information, comprising: Converting the state distribution of the neuromorphic synaptic array into a grayscale image via logarithmic mapping; The motion area is divided in the grayscale image by comparing with a preset threshold, and a binary mask representing the temporal motion clue is generated by combining 8-neighborhood connectivity analysis and morphological dilation.

8. The memristor-based motion information extraction and fast optical flow calculation system according to claim 7, characterized in that: The signal processing module performs optical flow calculation on the area corresponding to the temporal motion clue, including: The binary mask marked area of ​​temporal motion clues is used as the region of interest; Select the corresponding optical flow calculation method based on the application scenario, perform optical flow calculation using the region of interest, and obtain two-dimensional optical flow information; The two-dimensional optical flow information is combined with the binary mask to form a three-dimensional neuromorphic optical flow.

Images