Efficient data processing method applied to low-power-consumption hardware

Abstract

The invention discloses an efficient data processing method applied to low-power-consumption hardware. The efficient data processing method comprises the following steps: S1, receiving event stream data transmitted on a Speck chip through an SPI DMA interface; s2, triggering frame accumulation of the data through a soft interrupt mechanism; s3, performing corner detection on the received event stream one by one, and identifying key feature points in the image; s4, performing optical flow estimation on the event which is determined as the angular point by adopting a VSA method, and extracting motion information; and S5, transmitting the event stream, the calculated angular point data and the optical stream data to an upper computer through a Micro USB 2.0 interface. According to the scheme, based on an STM32L433 low-power-consumption microcontroller, data transmission is carried out through a Micro USB 2.0 interface, and edge deployment of optical flow estimation is achieved in combination with a hardware-friendly VSA optical flow algorithm. Under limited power consumption and computing resources, an optical flow estimation solution with high efficiency and precision is provided, and the actual requirement of a low-power-consumption embedded system for edge computing is met.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an efficient data processing method applied to low-power hardware. Background Art

[0002] In recent years, event cameras have attracted increasing attention as a new type of sensor. Unlike traditional frame image cameras, event cameras are biomimetic sensors that capture the brightness changes of each pixel asynchronously and output information about these changes in the form of an event stream, including the time, location, and sign of the brightness change. Compared with traditional cameras, event cameras have many advantages, such as ultra-high temporal resolution (millisecond level), high dynamic range, low power consumption, and high pixel bandwidth, which reduces motion blur. Therefore, in scenarios with high speed, low latency, and high dynamic range, event cameras have shown great potential in robotics and computer vision. However, because the data characteristics of event cameras are completely different from those of traditional cameras, traditional optical flow estimation methods cannot be directly applied to event data.

[0003] Currently, optical flow estimation algorithms for event data have made considerable progress and can be broadly categorized into two types: traditional methods and learning-based methods. Traditional algorithms typically rely on the principles and characteristics of event data to infer optical flow information. A classic example is the Lucas-Kanade (LK) optical flow method, which calculates optical flow by solving the optical flow equation based on the assumptions of constant brightness, minimal motion, and spatial consistency. However, this method is limited in its applicability and is generally effective only at corners or in areas with large gradient variations. It performs poorly in smooth and edge regions. Another representative traditional method is the contrast maximization algorithm, which maximizes an objective function to maximize the sharpest image formed when the event is transformed into a reference frame. Intuitively, this method estimates motion by restoring a sharp edge image of the event (i.e., motion compensation). However, this method is prone to event collapse.

[0004] The second category of algorithms is deep learning-based methods that estimate optical flow by leveraging potential correlations in the data or combining the aforementioned traditional principles. A major challenge facing event-based optical flow estimation is the current lack of sufficient real optical flow datasets, making it difficult to effectively evaluate and compare these methods, especially in supervised learning training. Although some standard datasets provide real optical flow information, these data often rely on depth sensors and traditional cameras, which limits their field of view and resolution and does not fully match the data characteristics of event cameras.

[0005] When deploying event optical flow estimation methods on low-power and low-performance hardware, traditional algorithms and deep learning-based methods each have some significant shortcomings, mainly reflected in computational complexity, hardware resource requirements, and real-time performance.

[0006] 1. Disadvantages of Traditional Algorithms: Traditional event optical flow estimation algorithms, such as the Lucas-Kanade (LK) optical flow method and the contrast maximization algorithm, can effectively estimate optical flow in certain application scenarios. However, their deployment on low-power and low-performance hardware has the following problems:

[0007] High computational complexity: Although these algorithms are relatively simple, they often require extensive matrix calculations, gradient estimation, and solving optical flow equations when processing event data streams. For example, the LK optical flow method typically requires calculating gradients and time-based local features around each pixel, which can be computationally expensive for hardware with limited computing resources.

[0008] High resource consumption: Traditional algorithms require high computational precision and large storage space to process event streams, especially in scenarios with high dynamic range and high-speed motion. Limited memory bandwidth and computing power on low-performance hardware can make these algorithms unable to effectively balance real-time performance and accuracy.

[0009] Poor real-time performance: In applications with high real-time processing requirements, the calculation process of traditional algorithms may lead to increased latency due to their requirements for time and space complexity, making it difficult to meet low-latency application requirements.

[0010] 2. Disadvantages of deep learning-based algorithms;

[0011] Deep learning-based event optical flow estimation algorithms have made significant progress in recent years, but they still face the following challenges when deployed on low-power and low-performance hardware:

[0012] High computing requirements: Deep learning methods typically require significant computing resources, especially convolutional neural networks (CNNs) or other complex neural network architectures. These networks involve extensive matrix operations and feature extraction when processing large amounts of event data. These calculations typically require powerful GPU support, while low-performance hardware often lacks the computing power to run these networks efficiently.

[0013] High storage requirements: Deep learning models often contain millions or even tens of millions of parameters, and the storage requirements of the models themselves are very high. On hardware with limited resources, storing these parameters not only requires a large amount of memory but can also cause data transmission bottlenecks, affecting the overall performance of the system.

[0014] Slow inference speed: While deep learning methods can improve the accuracy of optical flow estimation through end-to-end learning models, their inference process is often slow, especially on devices with limited computing power. For example, on embedded devices or low-power hardware, inference of deep learning models may require optimization or pruning to adapt to hardware resources. However, even optimized models may not be able to provide real-time optical flow estimation due to hardware limitations.

[0015] High energy consumption: Deep learning models require frequent floating-point calculations and large-scale data transfer during inference, resulting in relatively high power consumption. For hardware that needs to run for long periods of time and has limited power consumption (such as embedded devices, drones, or robots), the high energy consumption of deep learning models can become a major bottleneck in system design.

[0016] While traditional algorithms are relatively efficient in simple scenarios, their computational complexity and memory requirements remain high when processing event data, especially for complex scenarios such as rapid motion and occlusion, making it difficult to meet the real-time requirements of low-performance hardware. Deep learning-based algorithms, while superior in accuracy to traditional methods, have significantly higher computational and storage requirements and often require specialized hardware acceleration (such as GPUs or TPUs), which is often an unbearable burden for low-power, low-performance hardware platforms. Therefore, when deploying event optical flow estimation on low-power, low-performance hardware, how to reduce the demand for computational and storage resources while maintaining sufficient accuracy is an urgent challenge that needs to be addressed. Summary of the Invention

[0017] The object of the present invention is to overcome the deficiencies in the prior art and provide a kind of data processing method that is efficient and applied to low-power hardware. This scheme is based on STM32L433 low-power microcontroller, utilizes Micro USB 2.0 interface for data transmission, and combines hardware-friendly vector symbol architecture (VSA) optical flow algorithm to realize edge deployment of optical flow estimation. Under limited power consumption and computing resources, the present invention provides a kind of optical flow estimation solution with both high efficiency and precision, which meets the actual demand of low-power embedded system for edge computing.

[0018] The object of the present invention is achieved through the following technical solution: an efficient data processing method applied to low-power hardware, comprising the following steps:

[0019] S1 and the MCU receive event stream data transmitted from the Speck chip through the SPIDMA interface. The event stream data consists of 16 bits, where bits 1-7 are the X coordinates corresponding to the light intensity change, bits 8-14 are the Y coordinates corresponding to the light intensity change, bit 15 is the polarity direction of the light intensity change, and bit 16 is reserved and unused.

[0020] S2. Triggering frame accumulation of image data through a soft interrupt mechanism: Performing a logical "OR" operation on the pixels of the image; dividing the image into N×N blocks of equal size. When all pixel values in any block are 1, the current image accumulation is determined to be complete;

[0021] S3, perform corner detection on the received event stream one by one to identify key feature points in the image;

[0022] Extract the X and Y coordinates from the event stream, detect the adjacent area around the coordinates, and calculate the maximum number of consecutive pixel values 1 in the peripheral pixels with a radius of R around the coordinate point. When this value is within the specified threshold, the corner detection with a radius of R is considered to have passed.

[0023] Calculate the maximum number of OuterDistance pixels with a continuous value of 1 in the outer area with a radius of R+1. When this value is within the specified threshold, the corner point detection with a radius of R+1 is considered to have passed.

[0024] When the pixel value detection of the edges of the two radius areas passes, further direction detection is performed. The direction detection is constrained by the following formula:

[0025] InnerDistance*MaxOuterDistance <OuterDistance*MaxInnerDistanc

[0026] Among them, MaxInnerDistance and MaxOuterDistance refer to the total number of pixels in the area with a radius of R and R+1 around the coordinate point respectively; if the above constraints are met, it is determined that the directionality test has passed;

[0027] If both the pixel value detection and directionality detection of the received event pass, the event is determined to be a corner event; the 16th bit of its data structure is set to 1, and the event will be used as input to participate in the subsequent optical flow estimation detection; otherwise, the 16th bit of its data structure is set to 0; the event will not participate in the subsequent optical flow estimation detection;

[0028] S4, using the VSA method to estimate the optical flow of the event that has been determined as a corner point and extract motion information; combining the corner point event image with its adjacent frame images to estimate the optical flow;

[0029] For a transmitted corner event, quality detection is first performed, with the following process:

[0030] (1) Calculate whether the number of pixels with a value of 1 in the area with a radius of R1 and centered at the event flow coordinate point in the corner event image is greater than a preset threshold;

[0031] (2) Calculate whether the number of pixels with a value of 1 in the area with a radius of R2, centered at the event stream coordinate point, in adjacent frame images is greater than a preset threshold; R1 and R2 can be selected arbitrarily, preferably satisfying the following conditions: R1 is greater than the size of R in S3; R2 is greater than 4*R1;

[0032] (3) Calculate the VSA vector of each pixel in the area with a radius of R1 centered at the event stream coordinate point in each of the two images, and merge all the VSA vectors in the two images into a single VSA vector using the Majority Rule; determine whether the Hamming distance between the VSA vectors of the two images is less than a preset threshold;

[0033] If all three conditions are met, the quality check passes, the optical flow is true, and the optical flow estimation is performed on the event. Otherwise, the optical flow is false, the event is skipped, and the next event is processed. The process of optical flow estimation is as follows:

[0034] (1) Calculate the VSA vector corresponding to each pixel (i, j, k) in the area with a radius of R1, centered on the event flow coordinates, in the corner event image, and merge all vectors into a single VSA vector H1 using the Majority Rule;

[0035] (2) Calculate the VSA vector matrix of the area with radius R2 in the adjacent frame image, with the event flow coordinate as the center. Take a square area with radius R1 and slide it from left to right and from top to bottom in R2; each time it slides to an area, calculate the corresponding VSA vector H of the area according to the method of process (1). ij , where i and j correspond to the relative coordinates of the center of the sliding area corresponding to R2; all VSA vectors H ij Form a VSA vector matrix;

[0036] (3) H1 and H ij Calculate the Hamming distance to form a distance matrix D;

[0037] (4) Process the results as follows: determine whether the Hamming distances in the distance matrix D are less than a preset threshold; if so, proceed to the next step; otherwise, skip the event and continue to process the next event;

[0038] The 2D coordinate (i, j) corresponding to the smallest distance value in D is taken as the end point of the optical flow vector. The starting point of the optical flow vector is the (x, y) coordinate of the event transmitted by the speck chip. (ix, jy) = (ΔX, ΔY) is the vector representation of the optical flow;

[0039] If there are n identical minimum distance values, the mean of these optical flow vectors ΔX and ΔY is taken as the optical flow vector result for subsequent transmission;

[0040] S5, transmit the event stream and calculated corner point data and optical flow data to the host computer through the Micro USB 2.0 interface;

[0041] After completing the optical flow estimation and calculating the corresponding optical flow data ΔX and ΔY, the event stream, calculated corner point data, and optical flow data are transmitted to the host computer via the Micro USB 2.0 interface. The specific transmission method is as follows:

[0042] When the 16-bit event in the event stream is corner data, the next 16 bits must be optical flow data; when the 16-bit event in the event stream is not corner data, the next 16 bits are the next set of corner data or event data;

[0043] If the optical flow is true, the upper 8 bits of the optical flow data represent ΔY and the lower 8 bits represent ΔX; if the optical flow is false, the optical flow data is 0xFFFF.

[0044] The beneficial effects of the present invention are:

[0045] 1. Low-power implementation: Since the STM32L433 is an ultra-low-power MCU, the optimization of the present invention enables it to implement complex optical flow estimation functions in resource-constrained environments, which is suitable for mobile devices and edge computing scenarios.

[0046] 2. High resource efficiency: Fully utilize the 32-bit architecture and limited memory resources of the STM32L433, avoid the use of high-performance processors, and achieve a cost-effective technical solution.

[0047] 3. Wide adaptability: The algorithm and optimization strategy of the present invention are hardware-friendly and can be easily ported to other embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the data processing method of the present invention;

[0049] Figure 2 Schematic diagram of the low-power hardware structure of the present invention;

[0050] Figure 3 Schematic diagram of the hardware execution sequence of the main functions in the MCU of the present invention;

[0051] Figure 4 A schematic diagram of data frame accumulation according to the present invention;

[0052] Figure 5 This is a flowchart of the present invention for detecting corner points one by one;

[0053] Figure 6 Graph showing the construction method of the VSA vector of the present invention;

[0054] Figure 7 This is the process of Majority Rule operation of the present invention. DETAILED DESCRIPTION

[0055] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0056] like Figure 1 As shown, an efficient data processing method applied to low-power hardware of the present invention includes the following steps:

[0057] S1. MCU receives event stream data transmitted from the Speck chip through the SPIDMA interface.

[0058] like Figure 2 As shown in the figure, when the external light intensity changes, the light-sensing unit (Vibration of Light) on the Speck chip is triggered. Based on the relative position of the light intensity change, the Speck chip generates an event stream output. The data structure of the event stream is defined as follows (see the Event Flow in the figure): The event stream data consists of 16 bits, where bits 1-7 are the X coordinates corresponding to the light intensity change, bits 8-14 are the Y coordinates corresponding to the light intensity change, bit 15 is the polarity direction of the light intensity change, and bit 16 is a reserved bit and is not used;

[0059] The event stream is received via the SPI interface in DMA mode and stored in a specified buffer (BUFFER). When the number of events in the BUFFER reaches a set threshold, the SPI module triggers an interrupt and updates a flag, waiting for subsequent data processing operations.

[0060] Figure 3 This figure illustrates the hardware execution sequence of key functions within a microcontroller (MCU). The gray area in the figure represents the SPI triggering process per unit time, while the green arrows indicate how the SPI triggers a GPIO software interrupt via an interrupt function. An accumulation framing strategy is implemented in the GPIO interrupt service routine to meet real-time requirements.

[0061] The yellow and purple areas represent corner detection and optical flow estimation, respectively. These two operations are performed in the main loop (while loop) and have the lowest priority.

[0062] The design of integrating the accumulation framing strategy into the GPIO interrupt service routine aims to improve the real-time performance of image accumulation, enabling the system to quickly complete image accumulation and update after receiving the event stream, providing an efficient data foundation for subsequent corner detection and optical flow estimation.

[0063] The reason why the accumulation framing function is written into the GPIO interrupt is to better achieve the real-time performance of the accumulated images.

[0064] S2, triggering data frame accumulation through the soft interrupt mechanism, such as Figure 4 As shown, the specific process is as follows:

[0065] S2-1. Initialize three pictures t0, t1, and t2. The three pictures are stored in 128*128*1-bit arrays respectively. Initialize the pointers corresponding to t0, t1, and t2 to Δt0, Δt1, and Δt2 respectively.

[0066] S2-2, the picture corresponding to the Δt0 pointer receives the event stream data transmitted on the Speck chip and performs frame accumulation;

[0067] S2-3, perform a logical "OR" operation on the pixel points of the image corresponding to the Δt0 pointer to gradually complete the spatial accumulation of events;

[0068] S2-4, dividing the image into N×N blocks of equal size, and when all pixel values in any block are 1, it is determined that the current image accumulation is complete;

[0069] S2-5. After the accumulation is completed, the three pointers Δt0, Δt1, and Δt2 are rotated and exchanged with the three pictures to start a new round of picture accumulation operation. After the rotation and exchange, the Δt1 pointer points to the original picture of t0, the Δt2 pointer points to the original picture of t1, and the Δt0 pointer points to the original picture of t2. The picture corresponding to the Δt2 pointer is cleared. Because the array corresponding to the Δt2 pointer contains picture data after several operations, it must be cleared each time before receiving data again.

[0070] This process effectively utilizes the spatial distribution of event stream data through a block accumulation strategy, providing efficient and real-time input for subsequent corner detection and optical flow estimation.

[0071] S3, perform corner detection on the received event stream one by one to identify key feature points in the image;

[0072] Extract the X and Y coordinates from the event stream and detect the adjacent area around the coordinates. In the present invention, the peripheral pixel areas with radii R and R+1 around the coordinate point in the image (t0) corresponding to the Δt1 pointer are analyzed respectively: Through continuity detection, the maximum number of consecutive pixel values 1 in the peripheral pixels with a radius of R around the coordinate point is calculated. When this value is within the specified threshold range, it is considered that the corner point with a radius of R has passed the detection; the threshold range obeys the following formula:

[0073] InnerDistance∈[BN,TN]∪[MaxPixels-TN,MaxPixels-BN]

[0074] BN=R+1

[0075] TN=4*R+1

[0076] Among them, MaxPixels is the total number of peripheral pixels with a radius of R (side length of 2R+1) around the coordinate point; the peripheral pixels with a radius of R are defined as follows: the area with a radius of R corresponds to a square area with a side length of 2R+1, minus the area with a radius of R-1 corresponding to a square area with a side length of 2(R-1)+1, and the remaining area is the pixels.

[0077] Then, through continuity detection, the maximum number of OuterDistance of consecutive pixels with a value of 1 in the peripheral area with a radius of R+1 is calculated. When this value is within the specified threshold range, it is considered that the corner point detection with a radius of R+1 has passed. In this embodiment, the area with a radius of R 1 is taken as an example. Figure 5 As shown, when an event is detected (the red area in the figure), its coordinates are determined and the pixels of the 8 outer pixels with a radius of 1 are detected, which are 01011010 respectively. Then the maximum number of consecutive pixel values 1 in the outer area, InnerDistance = 2 (shown in the dotted box in the figure).

[0078] When the pixel value detection at the edge of the two radius areas passes, further direction detection is performed to ensure that the continuous pixel values of the inner and outer radii are consistent with the direction of the detection event. The direction detection is constrained by the following formula:

[0079] InnerDistance*MaxOuterDistance <OuterDistance*MaxInnerDistanc

[0080] Among them, MaxInnerDistance and MaxOuterDistance refer to the total number of pixels in the area with a radius of R and R+1 around the coordinate point respectively; if the above constraints are met, it is determined that the directionality test has passed;

[0081] If the received event pixel value detection and directionality detection pass, the event is determined to be a corner event; the 16th bit of its data structure is set to 1, indicating that it is a corner event; the event will be used as input for subsequent optical flow estimation detection. Otherwise, the 16th bit of its data structure is set to 0; the event will not be used for subsequent optical flow estimation detection.

[0082] S4. Use the VSA (Vectors Symbolic Architecture) method to perform optical flow estimation on the events that have been determined to be corner points and extract motion information. Optical flow estimation requires at least two frames of pictures in different time periods before and after, and the optical flow estimation is performed in combination with the picture that has been determined to be the corner point event and the adjacent frames of the picture. In this embodiment, the picture (array) corresponding to the Δt1 pointer has been determined to be the corner point time, and the picture (array) corresponding to the Δt2 pointer is its adjacent frame. Therefore, the optical flow estimation is performed in combination with the picture corresponding to the Δt1 pointer and the picture corresponding to the Δt2 pointer.

[0083] VSA has the following significant features:

[0084] (1) Simplified computational operations: Vector computations under the VSA framework only include XOR operations and majority rules, which are very efficient and easy to implement in hardware.

[0085] (2) Efficient high-dimensional vector structure: The high-dimensional vectors that make up VSA are composed only of 0s and 1s, which makes VSA calculations very hardware-friendly, especially in low-power devices, and can effectively save computing resources and energy.

[0086] like Figure 6 As shown, the VSA vector is constructed as follows:

[0087] First, two vectors L1 and L are randomly generated in the high-dimensional space. m , where m is the number of vectors to be generated; then, in L1 and L m The remaining intermediate vectors are generated between them; the generated intermediate vectors are constructed from left to right, and the bit of each intermediate vector will randomly inherit the vector L1 or L m Specifically, the number of inherited bits is inversely proportional to the distance between the intermediate vector and the starting vector L1; that is, the farther the intermediate vector is from L1, the fewer bits it inherits. m The more bits, the more constrained it is. It is constrained by the following formula:

[0088]

[0089] Where l is the intermediate vector L l The position of all vectors, for example, the position of vector L4 is 4. P is the generating vector L l The proportion of L1 is inherited, and the remaining 1-P proportion elements inherit L m .

[0090] Therefore, for an X*Y*Z image, where X is the image width, Y is the image height, and Z is the maximum grayscale value of the image, the above VSA vector construction method is used to generate X width vectors, Y height vectors, and Z grayscale value vectors. For the image pixel (i, j, k) with coordinates (i, j) and pixel value k, its VSA combination vector is equal to X i XOR Y j XOR Z k , where XOR stands for bitwise exclusive OR.

[0091] In the figure, 0000 is L1 and 1100 is L m , 1010 is the random inheritance of L1 and L m The generated intermediate vector.

[0092] In order to efficiently implement the Majority Rule in hardware and fully utilize the 32-bit parallelism of the MCU, a new efficient and easy-to-implement full adder majority principle method is proposed; the specific process is shown in the figure:

[0093] like Figure 7 As shown in the figure, the Majority Rule operation process is as follows: Assume that there are three VSA combination vectors A, B and C, which represent three different pixel points respectively. If you want to merge the features of the three combination vectors into one VSA vector and facilitate the subsequent Hamming distance detection, first calculate their sum vector:

[0094] Sum vector = A⊕B⊕C

[0095] Next, calculate the carry vector:

[0096] Carry vector = (A∧B)∨(C∧(A⊕B)

[0097] Next, each bit is tested using the calculated sum vector and carry vector to determine whether the number of 1s in each bit is a majority, thereby executing the Majority Rule. For example, if A is 01, B is 10, and C is 11, their sum vector is 00 and their carry vector is 11. This means that the number of 1s in the first bit of A, B, and C is 2, and the number of 1s in the second bit is 2. Specifically, the number of 1s in the first bit of each of the three vectors is calculated, and the values of the carry vector and the first bit of the sum vector are taken. These values are combined to form 10, which represents the number of 1s in the first bit.

[0098] Traditional VSA calculations rely on high-dimensional vectors, typically using vectors with dimensions of 1,000 to 10,000, to leverage the law of large numbers to maintain accuracy. However, processing such high-dimensional vectors is infeasible on performance-constrained edge hardware. Therefore, in this invention, the law of large numbers is skipped during vector construction to reduce computational effort and memory usage.

[0099] According to the rules for VSA vector generation, in a square area with a radius of m and a pixel value range of 0 to L-1, the VSA vector meets the following conditions:

[0100]

[0101] Where δ(x i ,x j ) represents the Hamming distance between any two vectors i and j in the generated horizontal axis x vector group; δ(y i ,y j ),δ(l i ,l j

[0102] Represent the vertical axis y and the grayscale value l respectively. The above three formulas constrain the size of the Hamming distance of the VSA vectors in the same vector group and different vector groups.

[0103] Using these constraint formulas, we can calculate the minimum number of bits after XOR operations on different high-dimensional vectors. Assuming the dimension of the vector is m and the pixel value size is L-1, the minimum number of bits of the new vector after the XOR operation is:

[0104] 64m 2 (L-1)

[0105] Among them, half of the vector length is randomly generated to provide enough bits for vertical redundancy in high-dimensional space. Theoretically, the minimum simplified vector size is:

[0106] 32m 2 (L-1)

[0107] This optimization significantly reduces computational and memory requirements while maintaining the effectiveness of VSA vectors, making it suitable for embedded hardware implementation.

[0108] For a transmitted corner event, quality detection is first performed, which mainly includes the following processes:

[0109] (1) Calculate whether the number of pixels with a value of 1 in the corner event image (the image corresponding to the Δt1 pointer) with the event flow coordinate point as the center and a radius of R1 is greater than a preset threshold;

[0110] (2) Calculate whether the number of pixels with a value of 1 in the adjacent frame image (the image corresponding to the Δt2 pointer) in the area with a radius of R2 centered at the event stream coordinate point is greater than a preset threshold; R1 and R2 can be selected arbitrarily, preferably satisfying the following conditions: R1 is greater than the size of R in S3; R2 is greater than 4*R1;

[0111] (3) Calculate the VSA vector of each pixel in the area with a radius of R1 centered at the event stream coordinate point in each of the two images, and merge all the VSA vectors in the two images into a single VSA vector using the Majority Rule; determine whether the Hamming distance between the VSA vectors of the two images is less than a preset threshold;

[0112] Through the detection of (1) and (2), we can determine whether the background is empty and does not have too many pixels. If so, the event is skipped. Through the detection of (3), we can determine whether the detection areas of the Δt2 image and the Δt1 image are similar. If they are too similar, it means there is a problem and the event is skipped.

[0113] If all three conditions are met, the quality check passes, the optical flow is true, and the optical flow estimation is performed on the event. Otherwise, the optical flow is false, the event is skipped, and the next event is processed. The process of optical flow estimation is as follows:

[0114] (1) Calculate the VSA vector corresponding to each pixel (i, j, k) in the corner event image (the image corresponding to the Δt1 pointer) with the event flow coordinate as the center and the radius R1, and merge all vectors into a single VSA vector H1 through the Majority Rule;

[0115] (2) Calculate the VSA vector matrix of the area with radius R2, centered on the event stream coordinates, in the adjacent frame image (the image corresponding to the Δt2 pointer). Take a square area with a radius of R1 and slide it from left to right and from top to bottom in R2. When sliding to an area, calculate the corresponding VSA vector H according to the method of process (1). ij , where i and j correspond to the relative coordinates of the center of the sliding area corresponding to R2; all VSA vectors H ij Form a VSA vector matrix;

[0116] (3) H1 and H ij Calculate the Hamming distance to form a distance matrix D;

[0117] (4) Process the results as follows: determine whether the Hamming distances in the distance matrix D are less than a preset threshold; if so, proceed to the next step; otherwise, skip the event and continue to process the next event;

[0118] The 2D coordinate (i, j) corresponding to the smallest distance value in D is taken as the end point of the optical flow vector. The starting point of the optical flow vector is the (x, y) coordinate of the event transmitted by the speck chip. (ix, jy) = (ΔX, ΔY) is the vector representation of the optical flow;

[0119] If there are n identical minimum distance values, the average of these optical flow vectors ΔX and ΔY is taken as the optical flow vector result for subsequent transmission.

[0120] S5, transmit the event stream and calculated corner point data and optical flow data to the host computer through the Micro USB 2.0 interface;

[0121] After completing the optical flow estimation and calculating the corresponding optical flow data ΔX and ΔY, the event stream, calculated corner point data, and optical flow data are transmitted to the host computer via the Micro USB 2.0 interface. The specific transmission method is as follows:

[0122] When the 16-bit event in the event stream is corner data, the next 16 bits must be optical flow data; when the 16-bit event in the event stream is not corner data, the next 16 bits are the next set of corner data or event data;

[0123] If the optical flow is true (if both corner detection and optical flow estimation pass, the optical flow is true; if it passes corner detection but fails S4 detection, the optical flow is false), the upper 8 bits of the optical flow data represent ΔY and the lower 8 bits represent ΔX; if the optical flow is false, the optical flow data is 0xFFFF.

[0124] When receiving event 0x000F, if the event is not a corner point, the Micro USB 2.0 interface transmits 0x000F to the host computer. If the event is a corner point but not optical flow data, bit 16 is set to 1, and the Micro USB 2.0 interface continuously transmits 0x800F and 0xFFFF to the host computer. 0xFFFF will be discarded by the host computer. If the event is a corner point and optical flow data, bit 16 is set to 1, and the Micro USB 2.0 interface continuously transmits 0x800F and 0xXXXX (optical flow data) to the host computer. 0xXXXX will be received by the host computer.

[0125] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

[0126] This invention successfully integrates framing, corner detection, and VSA-based optical flow estimation on an STM32L433 microcontroller through optimization of event stream processing and hardware design. The following deduces the technical effects of the technical solution from various aspects:

[0127] 1. Accumulation Framing: This invention adopts an efficient event stream accumulation framing strategy, combining an event triggering mechanism with regionalized pixel value accumulation. By limiting the accumulation region size and pixel value range (0 or 1), the complex pixel value calculations during the accumulation process are reduced. Accumulation is performed only on the event triggering region, reducing the use of memory and CPU computing resources. This allows the accumulation process to be completed in milliseconds at the 80MHz main frequency of the STM32L433, meeting the real-time requirements of high-frequency event streams.

[0128] 2. Corner Detection: Based on accumulated framing, this paper designs a directional-constrained corner detection process for images with all-one pixel values. This process uses directional constraints on inner and outer circle pixels to enhance corner detection accuracy. This avoids complex gradient calculations and employs simple rules based on event stream characteristics. Directional constraints effectively filter out noisy corners, improving corner detection accuracy. The detection process relies on simple logical operations, making it suitable for resource-limited MCUs such as the STM32L433.

[0129] 3. Optical flow detection based on VSA: The present invention adopts the VSA framework for optical flow detection and optimizes the hardware implementation by designing a vector inheritance strategy and an efficient Majority Rule calculation process: using the vector inheritance strategy to generate high-dimensional vectors, reducing redundant calculations and lowering memory usage. The introduction of the Majority Rule algorithm based on a full adder fully utilizes the 32-bit parallelism of the STM32 to significantly improve the calculation speed. The computational complexity of XOR and Majority Rule is extremely low and can be efficiently implemented on an MCU. A 32-bit VSA vector is generated through a vector inheritance strategy to meet hardware memory limitations. The accuracy of the optical flow estimation results is ensured by designing the Hamming distance matrix and threshold filtering.

[0130] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. An efficient data processing method applied to low-power hardware, characterized in that: It includes the following steps: S1. The MCU receives the event stream data transmitted from the Speck chip through the SPI DMA interface; the event stream data includes 16 bits. Among them, bits 1-7 are the X coordinates corresponding to the light intensity change, bits 8-14 are the Y coordinates corresponding to the light intensity change, bit 15 is the polarity direction of the light intensity change, and bit 16 is a reserved bit and not used; S2. Trigger the frame accumulation of the picture data through the soft interrupt mechanism: perform a logical "OR 1" operation on the pixel points of the picture; then divide the picture into N×N area blocks of the same size. When all pixel values in any one area block are 1, it is determined that the current picture accumulation is completed; S3. Perform corner detection one by one on the received event stream to identify the key feature points in the image; Extract the X and Y coordinates from the event stream, detect the adjacent area around the coordinates, and calculate the maximum number InnerDistance of consecutive pixel values of 1 among the outer pixel points with a radius of R around the coordinate point. When this value is within the specified threshold range, it is considered that the corner detection with a radius of R passes; Calculate the maximum number OuterDistance of consecutive pixel values of 1 in the outer area with a radius of R+1 around. When this value is within the specified threshold range, it is considered that the corner detection with a radius of R+1 passes; When the pixel value detections at the edges of the two radius areas both pass, further perform the directionality detection. The directionality detection is constrained by the following formula: InnerDistance*MaxOuterDistance<OuterDistance*MaxInnerDistanc where MaxInnerDistance and MaxOuterDistance respectively represent the total number of pixels in the areas with radii of R and R+1 around the coordinate point; if the above constraint is satisfied, it is determined that the directionality detection passes; If both the received event pixel value detection and the directionality detection pass, it is determined that the event is a corner event; set the 16th bit of its data structure to 1, and this event will be used as an input to participate in the subsequent optical flow estimation detection; otherwise, set the 16th bit of its data structure to 0; this event does not participate in the subsequent optical flow estimation detection; S4. Perform optical flow estimation on the events determined to be corners using the VSA method to extract motion information; combine the corner event image with its adjacent frame image for optical flow estimation; For a transmitted corner event, first perform quality detection, and the following process is available: (1) Calculate whether the number of pixel values of 1 in the area with a radius of R1 centered on the event stream coordinate point in the corner event image is greater than the preset threshold; (2) Calculate whether the number of pixel values of 1 in the area with a radius of R2 centered on the event stream coordinate point in the adjacent frame image is greater than the preset threshold; R1 is greater than R in S3, and R2 is greater than 4*R1; (3) Calculate the VSA vector of each pixel in the area with a radius of R1 centered at the event stream coordinate point in each of the two images, and merge all the VSA vectors in the two images into a single VSA vector using the Majority Rule; determine whether the Hamming distance between the VSA vectors of the two images is less than a preset threshold; If all three conditions are met, the quality check passes, the optical flow is true, and the optical flow estimation is performed on the event. Otherwise, the optical flow is false, the event is skipped, and the next event is processed. The process of optical flow estimation is as follows: (1) Calculate the VSA vector corresponding to each pixel (i, j, k) in the area with a radius of R1, centered on the event flow coordinates, in the corner event image, and merge all vectors into a single VSA vector H1 using the Majority Rule; (2) Calculate the VSA vector matrix of the area with radius R2 in the adjacent frame image, with the event flow coordinate as the center. Take a square area with radius R1 and slide it from left to right and from top to bottom in R2; each time it slides to an area, calculate the corresponding VSA vector H of the area according to the method of process (1). ij , where i and j correspond to the relative coordinates of the center of the sliding area corresponding to R2; all VSA vectors H ij Form a VSA vector matrix; (3) H1 and H ij Calculate the Hamming distance to form a distance matrix D; (4) Process the results as follows: determine whether the Hamming distances in the distance matrix D are less than a preset threshold; if so, proceed to the next step; otherwise, skip the event and continue to process the next event; The 2D coordinate (i, j) corresponding to the smallest distance value in D is taken as the end point of the optical flow vector. The starting point of the optical flow vector is the (x, y) coordinate of the event transmitted by the speck chip. (ix, jy) = (ΔX, ΔY) is the vector representation of the optical flow; If there are n identical minimum distance values, the mean of these optical flow vectors ΔX and ΔY is taken as the optical flow vector result for subsequent transmission; S5, transmit the event stream and calculated corner point data and optical flow data to the host computer through the Micro USB 2.0 interface; After completing the optical flow estimation and calculating the corresponding optical flow data ΔX and ΔY, the event stream, calculated corner point data, and optical flow data are transmitted to the host computer via the Micro USB 2.0 interface. The specific transmission method is as follows: When the 16-bit event in the event stream is corner data, the next 16 bits must be optical flow data; when the 16-bit event in the event stream is not corner data, the next 16 bits are the next set of corner data or event data; If the optical flow is true, the upper 8 bits of the optical flow data represent ΔY and the lower 8 bits represent ΔX; if the optical flow is false, the optical flow data is 0xFFFF.

Images