A method and apparatus for motion compensated hardware acceleration

Abstract

The application relates to the technical field of brain-like perception and computation, in particular to a motion compensation hardware acceleration method and device, which comprises the following steps: denoising is performed on current events and past preset time window events in an event stream based on FPGA to obtain a denoised event stream; events in the denoised event stream are divided into a plurality of time windows; pixel positions of each event in each time window are superimposed with positions of camera motion based on IMU data to obtain a warped event stream; time window event accumulation is performed on the warped event stream; a CPU saves an obtained event accumulation frame in an off-chip RAM; average time stamps of each pixel of an image are obtained from the event accumulation frame; Gaussian filtering is performed to obtain a motion compensation frame, which is output under the control of the CPU; and the application can improve the efficiency of event camera motion compensation and reduce the delay.

Description

Technical Field

[0001] This invention relates to the field of neuromorphic perception and computing technology, specifically to a motion compensation hardware acceleration method and device. Background Technology

[0002] Event cameras are neuromorphic vision sensors inspired by biological vision, offering advantages such as high temporal resolution, high dynamic range (up to 140dB), and low power consumption. Unlike traditional cameras that capture images at a fixed frame rate, event cameras are sensitive to relative changes in brightness and output asynchronous event streams. They independently encode positive relative increases and negative relative decreases in light intensity, outputting events representing these changes. Therefore, event cameras perform exceptionally well in a variety of challenging applications, such as object detection, pose estimation, motion analysis, and SLAM. Event cameras simulate the imaging mechanism of the human eye by implementing a three-level abstraction of the photoreceptor layer, bipolar cell layer, and ganglion cell layer in the retinal structure.

[0003] Event cameras possess a unique visual perception mechanism that enables them to respond naturally to dynamic changes, making them ideal for high-speed target detection. However, when deployed on drones or intelligent robots, the camera's own motion generates a large number of background events, requiring powerful data processing capabilities and motion compensation to filter them out. When the accumulation time is short, they are susceptible to interference from background noise. High-performance central processing units (CPUs) alone are insufficient to meet these demanding real-time and data throughput requirements.

[0004] Many high-speed object detection methods that leverage the dynamic response characteristics of event cameras utilize the spatiotemporal similarity of continuous event streams generated by moving objects, employing clustering algorithms for effective clustering and classification. When deployed on drones or intelligent robots, the camera's own motion generates a large number of background events, thus requiring powerful data processing capabilities and motion compensation to filter these background events.

[0005] Current motion compensation algorithms require the collection of a batch of events and IMU data within a specified time. When the accumulated time is short, it is impossible to form an effective motion trajectory to compensate for background events, making it unsuitable for high frame rate monitoring. At the same time, motion compensation calculations at high detection frame rates place very high demands on the computational speed of unmanned systems. Due to the limited payload capacity and energy consumption of unmanned systems, high requirements are placed on the computational energy consumption and computational speed of the algorithms. Summary of the Invention

[0006] In view of the above problems, the present invention provides a motion compensation hardware acceleration method and apparatus, which solves the technical problems of high latency and low efficiency in motion compensation for event cameras in the prior art.

[0007] On one hand, the present invention provides a motion compensation hardware acceleration method, comprising the following steps:

[0008] Step S1: The CPU receives the event stream and IMU data output by the event camera and forwards them to the FPGA. The FPGA performs denoising based on the current event and past events within a preset time window in the event stream to obtain a denoised event stream.

[0009] Step S2: The FPGA divides the events in the denoised event stream into multiple time windows; based on the IMU data, the pixel positions of each event in each time window are superimposed with the camera motion position to obtain the warped event stream;

[0010] Step S3: The FPGA accumulates the warped event stream within a time window, and the CPU saves the accumulated event frames in off-chip RAM.

[0011] Step S4: The FPGA reads the event accumulation frame from the off-chip RAM and performs row buffering in the on-chip RAM to obtain the average timestamp of each pixel in the image; Gaussian filtering is performed based on the average timestamp to obtain the motion compensation frame and send it to the off-chip RAM.

[0012] Step S5: The CPU outputs the motion compensation frame from the external RAM as the compensation result.

[0013] Preferably, in step S1, the event stream records multiple events, and the events include: pixel coordinates, event timestamps, and event polarities; the IMU data includes angular velocity and timestamps.

[0014] Preferably, the step S1, in which the FPGA performs noise reduction based on the current event and past events within a preset time window in the event stream, specifically includes:

[0015] Determine the spatial neighborhood of the current event and the past duration is Events within the time window retain the same polarity as the current event. Each event is used as the denoised output, ultimately resulting in a denoised event stream;

[0016] During the judgment process, for the event's timestamp, save bits 5-12 of the timestamp.

[0017] Preferably, step S2 specifically includes:

[0018] Step S2-1: Use time windows with adjustable overlap rates to overlap and segment the event stream, with each time window having a corresponding event;

[0019] Step S2-2: Use the average angular velocity on the timestamps of events in the IMU data to obtain the rotation angle of the pixel coordinates. The rotation matrix of the pixel coordinates is calculated using the Rodriguez rotation formula;

[0020] Step S2-3: Reproject the pixel coordinates of each event within each time window according to the pre-stored camera intrinsic parameter matrix and the rotation matrix to obtain the warped event stream.

[0021] Preferably, in step S2-2, the sine and cosine in the Rodriguez rotation formula are calculated using a coordinate rotation digital computer algorithm;

[0022] In steps S2-3, a lookup table is used to store the camera intrinsic parameter matrix.

[0023] Preferably, step S3 specifically includes:

[0024] For the warp event stream, the FPGA performs the following processing within a preset time window: events occurring at pixels... The timestamps of all events are weighted and accumulated to obtain the cumulative timestamp frame. Count the total number of occurrences for each pixel and record it as an event counting frame. The accumulated timestamp frame and the event count frame together serve as the event accumulation frame, which is stored by the CPU in off-chip RAM.

[0025] Preferably, in step S4, the expression for calculating the average timestamp of each pixel in the image is:

[0026]

[0027] in, Indicates coordinates as The average timestamp of the pixels.

[0028] Preferably, in step S4, the step of performing Gaussian filtering based on the average timestamp to obtain the motion-compensated frame specifically includes:

[0029] (1) For each pixel, multiply the average timestamp by all non-repeating elements of the Gaussian matrix to obtain multiple product values ​​of the corresponding Gaussian kernel;

[0030] (2) For each pixel, the product values ​​of the corresponding Gaussian kernel are accumulated in the row direction, and stored in the row buffer after multiplexing; the column values ​​are read sequentially from the row buffer and accumulated to finally obtain the Gaussian filter values ​​of all pixels.

[0031] (3) Use a comparator to delete pixels whose average timestamp is greater than a preset threshold to obtain motion-compensated frames.

[0032] On one hand, the present invention provides a motion compensation hardware acceleration device, comprising:

[0033] The event stream preprocessing module enables the CPU to receive the event stream and IMU data output by the event camera and forward them to the FPGA. The FPGA performs denoising based on the current event and past events within a preset time window in the event stream to obtain a denoised event stream.

[0034] The affine transformation acceleration module enables the FPGA to divide the events in the denoised event stream into multiple time windows; based on IMU data, the pixel positions of each event in each time window are superimposed with the camera motion position to obtain the warped event stream;

[0035] The spatiotemporal information aggregation module enables the FPGA to accumulate warped event streams within a time window, and the CPU saves the accumulated event frames in off-chip RAM.

[0036] The background event filtering module is used to enable the FPGA to read the event accumulation frame from the off-chip RAM and perform row buffering in the on-chip RAM to obtain the average timestamp of each pixel in the image; and to perform Gaussian filtering based on the average timestamp to obtain motion compensation frames and send them to the off-chip RAM.

[0037] The output module enables the CPU to output motion-compensated frames from external RAM as compensation results.

[0038] Compared with the prior art, the present invention has at least the following beneficial effects:

[0039] (1) The motion compensation algorithm accelerator of the present invention processes the calculation processes such as event stream denoising, pixel position motion compensation, and event accumulation in parallel on the FPGA, and utilizes the efficient data caching mechanism of off-chip RAM and on-chip RAM to significantly improve the frame rate of the equivalent event frames generated by the event camera motion compensation algorithm. After the event stream data is forwarded from the CPU to the FPGA, the FPGA performs efficient data stream processing in steps such as event denoising, time window division, and pixel position compensation, so that motion compensation frames can be generated quickly to meet the high frame rate data requirements of subsequent target detection, recognition, and tracking algorithms.

[0040] (2) The present invention implements the motion compensation calculation process using FPGA. Through this heterogeneous division of labor, the power consumption of the overall system is greatly reduced, which is especially suitable for unmanned systems with strict requirements for energy consumption, and realizes long-term continuous monitoring and stable operation.

[0041] (3) The step design of this invention fully utilizes the hardware advantages of FPGA in timing processing and parallel event accumulation, thereby improving the calculation speed of the motion compensation process. With the help of efficient on-chip resource management and external high-speed cache, this invention improves the real-time performance of detection and response of unmanned systems, and better supports the real-time and fast processing requirements in practical applications. Attached Figure Description

[0042] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0043] Figure 1 A flowchart of the motion compensation hardware acceleration method provided by the present invention.

[0044] Figure 2 This is a top-level diagram of the motion compensation hardware acceleration method provided by the present invention.

[0045] Figure 3 A schematic diagram of the denoising algorithm provided by the present invention.

[0046] Figure 4 The schematic diagram shows the hardware acceleration method of the polarity-based denoising algorithm provided by this invention.

[0047] Figure 5 The schematic diagram of timestamp filtering provided for this invention.

[0048] Figure 6 This is a schematic diagram of the motion compensation hardware acceleration results provided by the present invention. Detailed Implementation

[0049] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0050] The motion compensation hardware acceleration method provided by this invention improves the effective frame rate of the motion compensation algorithm by coordinating preprocessing, time window division, motion parameter calculation, coordinate reprojection to spatiotemporal accumulation, and background culling, utilizing event frames based on spatiotemporal polarity consistency and overlapping window accumulation methods. Simultaneously, it utilizes off-chip RAM for data storage and implements a row buffer in on-chip RAM to ensure sufficient data access bandwidth, thus achieving accurate motion compensation at high frame rates in hardware.

[0051] like Figure 1 , Figure 2As shown, this invention discloses a motion compensation hardware acceleration method, comprising the following steps:

[0052] Step S1: The CPU receives the event stream and IMU data output by the event camera and forwards them to the FPGA. The FPGA performs denoising based on the current event and past events within a preset time window in the event stream to obtain a denoised event stream.

[0053] This invention combines a central processing unit (CPU) and a field-programmable gate array (FPGA) into a heterogeneous system-on-a-chip (SoC) to achieve high frame rate motion compensation for images captured by an event camera.

[0054] The event camera senses changes in light intensity of pixels within its field of view asynchronously, and outputs an event in real time when the light intensity change of a pixel exceeds a set threshold, forming an event stream.

[0055] The event includes the location coordinates of the pixel where the event occurred, the event timestamp, and the event polarity, where the event polarity indicates whether the light intensity is increasing (positive polarity) or decreasing (negative polarity). The event stream is output in the form of a data sequence with the above structure, achieving sparse, high temporal resolution capture of light intensity changes in dynamic scenes.

[0056] The event camera captures images while in motion. Internally, the event camera contains an Inertial Measurement Unit (IMU) that outputs real-time IMU data representing the camera's motion state. This IMU data includes acceleration, angular velocity, and timestamps. The IMU data allows for reprojection (warping) of events in subsequent steps, enabling events occurring at different times against a static background to be reprojected onto the same pixel.

[0057] In this step, the CPU receives the event stream and IMU data output by the event camera and forwards them to the FPGA. The FPGA performs noise reduction based on the current event and past events within a preset time window in the event stream, as described in detail below.

[0058] Noise is typically spatially and temporally incoherent in polarity. Therefore, the consistency of polarity in spacetime can be utilized for noise reduction. For example... Figure 3 As shown, the present invention preserves the spatial neighborhood of the current event. and the past duration is Within the time window The denoising output is obtained by taking events of the same polarity as the denoising output. This allows for processing on an event-by-event basis, is computationally simple, and has a high denoising rate, making it an effective denoising algorithm.

[0059] Hardware denoising requires storing the timestamps of the latest positive and negative events generated for each pixel. Due to the microsecond-level temporal resolution of the event camera, storing complete timestamps consumes significant memory resources. Considering the 30ms time window of the denoising algorithm and the monotonically increasing nature of timestamps, this invention preserves the spatial neighborhood of the current event. and the past duration is During an event within a time window, for each event timestamp, only bits 5-12 of the timestamp are saved, and only values ​​less than the current event timestamp are considered. Furthermore, adjacent pixels can be concatenated to improve data read bandwidth.

[0060] Through the above methods, the FPGA of this invention can complete the denoising of each event within 4 clock cycles, which can speed up the processing speed.

[0061] Step S2: The FPGA divides the denoised event stream into overlapping time windows, with each time window corresponding to a specific event; based on IMU data, the pixel positions of each event in each time window are superimposed with the camera motion position to obtain a warped event stream;

[0062] In this step, a time-series slice window with an adjustable overlap rate is used to overlap and segment the event stream, with each time window containing a corresponding event.

[0063] This invention uses IMU data to perform fast motion compensation on events in each time window. The IMU data is used to obtain the camera's motion, and the events are distorted based on this motion, so that events generated at different times in a static background are reprojected onto the same pixel, as described in detail below.

[0064] The rotation angle of the event pixel is obtained by using the average angular velocity of the IMU at the event's timestamp. The rotation matrix of the event pixel is calculated using the Rodriguez rotation formula, expressed as:

[0065]

[0066] in, R Represents the vector around the unit axis n Angle θ Rotation matrix of rotation, I As a unit array, n T for n transpose, Represents vector from the unit axis n A converter to an antisymmetric matrix.

[0067] This invention employs the CORDIC (Coordinate Rotation Digital Computer) algorithm to calculate sine and cosine. The CORDIC algorithm is an iterative method for calculating trigonometric functions (such as sine and cosine), hyperbolic functions, logarithmic functions, and other functions. The core idea of ​​the CORDIC algorithm is to decompose an angle into a series of smaller angles and gradually approach the target angle by rotating a vector. Each iteration approximates the target angle through a rotation of a fixed angle. Finally, after multiple iterations, the sine and cosine values ​​of the angle can be obtained. It approximates the calculation results through a series of simple shift and addition operations, making it very suitable for hardware implementation. This invention uses a parallel pipelined CORDIC algorithm, where each stage of the pipeline executes a specific iterative algorithm. The CORDIC algorithm gradually approximates the sine and cosine values ​​of the target angle through a series of simple shift and addition operations. Because the algorithm does not require multipliers, it is very suitable for application in hardware implementation.

[0068] Obtain the rotation matrix Subsequently, this invention establishes a mapping relationship from 3D rotation to the 2D image plane based on the pre-stored camera intrinsic parameter matrix, and uses a matrix multiplier to perform reprojection calculation on the event coordinates, expressed as:

[0069]

[0070] in, K This represents the camera intrinsic parameter matrix. K -1 express K The inverse matrix, p Indicates the pixel coordinates of the event. p’ This represents the converted event pixel coordinates.

[0071] In the above steps, the present invention pre-sets the camera intrinsic parameter matrix. and Saved as a lookup table, a matrix multiplier is used to perform a reprojection calculation once per clock cycle, thereby keeping the processing latency of each event at a low level, achieving a throughput of 1 cycle per event and a latency of 29 cycles.

[0072] Through the above methods, the present invention completes the process of reprojecting (or warping) events, superimposing the effect of camera motion on the pixel position of the event, so that events generated in a static background at different times are reprojected onto the same pixel. Through this motion compensation, the spatial dispersion of events caused by camera motion can be greatly alleviated, the spatial consistency and clarity of event images can be improved, and the latency can be reduced.

[0073] Figure 4A hardware processing framework diagram for polarity-based denoising is shown.

[0074] Step S3: The FPGA accumulates the warped event stream within a time window, and the CPU saves the accumulated event frames in off-chip RAM.

[0075] In this step, the event accumulation frame includes an accumulation timestamp frame and an event count frame. The FPGA receives and accumulates all events within a preset time window, and based on the event timestamp and pixel coordinate information, events occurring at the same pixel are counted. The timestamps of all events within the current time window are weighted and accumulated to obtain the cumulative timestamp frame for each pixel within that window. Simultaneously, the total number of events occurring at each pixel within that time window is counted and recorded as an event counting frame. .

[0076] The CPU writes and stores the above processing results in off-chip RAM, providing a data basis for subsequent steps.

[0077] Step S4: The FPGA reads the event accumulation frame from the off-chip RAM and performs row buffering in the on-chip RAM, then obtains the average timestamp of each pixel in the image; based on the average timestamp, it performs Gaussian filtering to obtain motion-compensated frames and sends them to the off-chip RAM.

[0078] For reading accumulated timestamp frames and event count frames from the on-chip RAM line buffer, this invention uses a timestamp filter hardware accelerator built on an FPGA to process them and obtain the average timestamp of the pixel, expressed as:

[0079]

[0080] Among them, , )and These represent the cumulative timestamp frame, event count frame, and coordinates, respectively. The average timestamp of the pixels.

[0081] Next, Gaussian filtering and comparison filtering are applied to the average timestamp of the pixels, such as... Figure 5 As shown, the specific steps include:

[0082] (1) For each pixel, the average timestamp is multiplied by all non-repeating elements of the Gaussian matrix to obtain multiple product values ​​of the corresponding Gaussian kernel. Due to the symmetry of the Gaussian filter kernel, this method can save DSP resources.

[0083] (2) For each pixel, the product values ​​of the corresponding Gaussian kernel are accumulated in the row direction, and stored in the row buffer after multiplexing; the column values ​​are read sequentially from the row buffer and accumulated to finally obtain the Gaussian filter values ​​of all pixels.

[0084] The expression for the above steps is:

[0085]

[0086] in Indicates that it has a spatial index Gaussian filter kernel, Representing coordinates Gaussian filter value at that point, denoted as the size of the Gaussian filter kernel.

[0087] (3) Use a comparator to delete pixels with an average timestamp greater than a preset threshold to obtain motion-compensated frames. The expression is:

[0088]

[0089] in, Represents pixel coordinates The threshold binary value indicates whether to retain the pixel at that location. For comparison thresholds.

[0090] In this invention, the input to the timestamp filter hardware accelerator is the accumulated timestamp frames and event count frames. The entire hardware accelerator calculates the average timestamp, performs Gaussian filtering and threshold filtering operations, and outputs the result after motion compensation.

[0091] The timestamp filter accelerator reads data line by line from the input line buffer. Since the calculation of each pixel is independent, parallel processing of the image is achieved by dividing the image into blocks of 8 pixels, which can reduce processing latency.

[0092] Step S5: The CPU outputs the motion compensation frame from the external RAM as the compensation result.

[0093] The CPU reads the motion compensation frame obtained and stored in step S4 from the off-chip RAM. Subsequently, the CPU sends the motion compensation frame to an external device or subsequent processing module through the data output interface to output the motion compensation result, thereby completing the entire event stream motion compensation process.

[0094] Figure 6 The results of motion compensation hardware acceleration are shown.

[0095] The present invention also provides a motion compensation hardware acceleration device, comprising the following modules:

[0096] The event stream preprocessing module enables the CPU to receive the event stream and IMU data output by the event camera and forward them to the FPGA. The FPGA performs denoising based on the current event and past events within a preset time window in the event stream to obtain a denoised event stream.

[0097] The affine transformation acceleration module enables the FPGA to divide the events in the denoised event stream into multiple time windows; based on IMU data, the pixel positions of each event in each time window are superimposed with the camera motion position to obtain the warped event stream;

[0098] The spatiotemporal information aggregation module enables the FPGA to accumulate warped event streams within a time window, and the CPU saves the accumulated event frames in off-chip RAM.

[0099] The background event filtering module is used to enable the FPGA to read the event accumulation frame from the off-chip RAM and perform row buffering in the on-chip RAM to obtain the average timestamp of each pixel in the image; and to perform Gaussian filtering based on the average timestamp to obtain motion compensation frames and send them to the off-chip RAM.

[0100] The output module enables the CPU to output motion-compensated frames from external RAM as compensation results.

[0101] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution of this invention is provided through a specific embodiment.

[0102] Example 1

[0103] This invention implements a motion compensation hardware acceleration method on the AVNET Ultra96 development board, powered by the Xilinx Zynq UltraScale+MPSoC ZU3EG. This chip integrates a 64-bit quad-core ARM Cortex-A53 processor and programmable logic units (PLUs). Its PLUs include 154,000 system logic units, comprising 70,560 lookup tables (LUTs), 141,120 flip-flops (FFs), and 432×18k block random access memory (BRAMs) with 7.6Mb of RAM and 360 digital signal processor (DSP) units. The final design is completed using high-level synthesis (HLS).

[0104] To simulate a real-world drone operating scenario, a DAVIS 346 event camera was mounted on a quadcopter, and a basketball was thrown in front of the drone. Nine datasets of this type of scenario were recorded to evaluate the algorithm's performance. The Shapes_6DOF dataset was used to capture synchronized frames and event data of multiple regularly shaped moving objects using a DAVIS camera. To establish quantitative evaluation metrics, the true location of the moving object was extracted from the high-fidelity frame sequence using a morphological edge detection algorithm, enabling a systematic comparison of the precision and recall of different methods.

[0105] The processing speed of the method of this invention was compared and analyzed in terms of computational throughput between FPGA-accelerated implementation and CPU-only execution of the complete algorithm. The processing speed of each component is shown in Table 1. Compared with the pure software implementation running on an Arm Cortex-A53 processor in Ultra96, the heterogeneous computing paradigm shows an overall speedup of 23.8 times. The limited speedup of denoising is related to sequential dependency constraints, while the motion compensation accelerator benefits extensively from the massive parallelization achieved by spatial architecture optimization.

[0106] Table 1

[0107]

[0108] Table 2 shows a comparison of resource consumption per module reported by Xilinx Vivado. LUTs and DSPs were observed to have the highest utilization. DSPs are primarily used for the multiplication and accumulation operations required in Gaussian filtering. Xilinx Vivado reports that logic modules consume approximately 2.895W. Compared to the denoising accelerator, the motion compensation accelerator consumes 3.25 times more power. This difference is fundamentally rooted in the different computational paradigms between the arithmetic-intensive multiplication and accumulation operations in motion compensation and the comparison-driven operational logic in denoising.

[0109] Table 2

[0110]

[0111] While the specific embodiments of the present invention depict actions or steps in a particular order, this should be understood as requiring such actions or steps to be performed in the shown specific order or sequential order, or requiring all illustrated actions or steps to be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations. The above descriptions are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention.

[0112] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A motion compensation hardware acceleration method, characterized in that, Includes the following steps: Step S1: The CPU receives the event stream and IMU data output by the event camera and forwards them to the FPGA. The FPGA performs denoising based on the current event and past events within a preset time window in the event stream to obtain a denoised event stream. Step S2: The FPGA divides the events in the denoised event stream into multiple time windows; The warped event stream is obtained by superimposing the pixel positions of each event in each time window with the camera motion position based on IMU data; Step S3: The FPGA accumulates the warped event stream within a time window, and the CPU saves the accumulated event frames in off-chip RAM. Step S4: The FPGA reads the event accumulation frame from the off-chip RAM and performs row buffering in the on-chip RAM to obtain the average timestamp of each pixel in the image; Gaussian filtering is performed based on the average timestamp to obtain motion-compensated frames, which are then sent to off-chip RAM. Step S5: The CPU outputs the motion compensation frame from the off-chip RAM as the compensation result; In step S4, the step of performing Gaussian filtering based on the average timestamp to obtain the motion-compensated frame specifically includes: (1) For each pixel, multiply the average timestamp by all non-repeating elements of the Gaussian matrix to obtain multiple product values ​​of the corresponding Gaussian kernel; (2) For each pixel, the product values ​​of the corresponding Gaussian kernel are accumulated in the row direction, and stored in the row buffer after multiplexing; the column values ​​are read sequentially from the row buffer and accumulated to finally obtain the Gaussian filter values ​​of all pixels. (3) Use a comparator to delete pixels whose average timestamp is greater than a preset threshold to obtain motion-compensated frames.

2. The motion compensation hardware acceleration method according to claim 1, characterized in that, In step S1, the event stream records multiple events, including pixel coordinates, event timestamps, and event polarities; the IMU data includes angular velocity and timestamps.

3. The motion compensation hardware acceleration method according to claim 2, characterized in that, The step S1, which involves denoising the FPGA based on the current event and past events within a preset time window in the event stream, specifically includes: Determine the spatial neighborhood of the current event and the past duration is Events within the time window retain the same polarity as the current event. Each event is used as the denoised output, ultimately resulting in a denoised event stream; During the judgment process, only bits 5-12 of the timestamp of the event are saved.

4. The motion compensation hardware acceleration method according to claim 3, characterized in that, Step S2 specifically includes: Step S2-1: Use time windows with adjustable overlap rates to overlap and segment the event stream, with each time window having a corresponding event; Step S2-2: Use the average angular velocity on the timestamps of events in the IMU data to obtain the rotation angle of the pixel coordinates. The rotation matrix of the pixel coordinates is calculated using the Rodriguez rotation formula; Step S2-3: Reproject the pixel coordinates of each event within each time window according to the pre-stored camera intrinsic parameter matrix and the rotation matrix to obtain the warped event stream.

5. The motion compensation hardware acceleration method according to claim 4, characterized in that, In step S2-2, the sine and cosine in the Rodriguez rotation formula are calculated using a coordinate rotation digital computer algorithm. In steps S2-3, a lookup table is used to store the camera intrinsic parameter matrix.

6. The motion compensation hardware acceleration method according to claim 5, characterized in that, Step S3 specifically includes: For the warp event stream, the FPGA performs the following processing within a preset time window: events occurring at pixels... The timestamps of all events are weighted and accumulated to obtain the cumulative timestamp frame. Count the total number of occurrences for each pixel and record it as an event counting frame. The accumulated timestamp frame and the event count frame together serve as the event accumulation frame, which is stored by the CPU in off-chip RAM.

7. The motion compensation hardware acceleration method according to claim 6, characterized in that, In step S4, the expression for calculating the average timestamp of each pixel in the image is: in, Indicates coordinates as The average timestamp of the pixels.

8. A motion compensation hardware acceleration device, characterized in that, include: The event stream preprocessing module enables the CPU to receive the event stream and IMU data output by the event camera and forward them to the FPGA. The FPGA performs denoising based on the current event and past events within a preset time window in the event stream to obtain a denoised event stream. The affine transformation acceleration module enables the FPGA to divide events in the denoised event stream into multiple time windows; The warped event stream is obtained by superimposing the pixel positions of each event in each time window with the camera motion position based on IMU data; The spatiotemporal information aggregation module enables the FPGA to accumulate warped event streams within a time window, and the CPU saves the accumulated event frames in off-chip RAM. The background event filtering module is used to enable the FPGA to read the event accumulation frame from the off-chip RAM and perform row buffering in the on-chip RAM to obtain the average timestamp of each pixel in the image; Gaussian filtering is performed based on the average timestamp to obtain motion-compensated frames, which are then sent to off-chip RAM. The output module enables the CPU to output motion-compensated frames from off-chip RAM as compensation results. The step of performing Gaussian filtering based on the average timestamp to obtain the motion-compensated frame specifically includes: (1) For each pixel, multiply the average timestamp by all non-repeating elements of the Gaussian matrix to obtain multiple product values ​​of the corresponding Gaussian kernel; (2) For each pixel, the product values ​​of the corresponding Gaussian kernel are accumulated in the row direction, and stored in the row buffer after multiplexing; the column values ​​are read sequentially from the row buffer and accumulated to finally obtain the Gaussian filter values ​​of all pixels. (3) Use a comparator to delete pixels whose average timestamp is greater than a preset threshold to obtain motion-compensated frames.

Images