3D reconstruction method based on binocular event stream matching

Abstract

The application provides a 3D reconstruction method based on binocular event stream matching, utilizes high time resolution and data sparsity of an event camera, and realizes 3D perception of a dynamic scene with ultra-low delay. Two synchronous event cameras are adopted, the method does not process image frames, but directly matches corresponding event streams from left and right cameras in space and time on a hardware level in real time. The matched event pairs directly generate a 3D point cloud with high refresh rate, and the point cloud only contains the changed or moving part in the scene. Through the architecture of "binocular event camera + hardware acceleration + real-time algorithm", the asynchronous characteristics of the event camera and the parallel computing capability of the hardware are fully utilized, and the 3D reconstruction of the dynamic scene with ultra-low delay, high accuracy and high sparsity is realized. Compared with the traditional method, the scheme has obvious advantages in the high-speed dynamic scene, and can be widely applied to the fields of industrial robots, automatic driving, virtual reality and the like.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, and in particular to a 3D reconstruction method based on binocular event flow matching, as well as a 3D reconstruction system, electronic device, and computer-readable storage medium based on binocular event flow matching. Background Technology

[0002] Traditional 3D reconstruction methods primarily rely on frame-based cameras (such as CCD / CMOS cameras) to generate 3D point clouds by capturing consecutive image frames and utilizing techniques like stereo vision or structured light. However, this method has a fatal flaw in high-speed dynamic scene perception:

[0003] Low temporal resolution: Frame-based cameras typically operate at 30-60fps (33-16.7ms interval between frames), which cannot capture details of high-speed moving objects (such as car collisions or robots picking up ping-pong balls), resulting in large 3D reconstruction delays (≥100ms), which cannot meet real-time control requirements (such as robot navigation requiring a delay of <100μs).

[0004] Large data redundancy: Frame-based cameras output the entire image (including static background), which is a huge amount of data (e.g., a 1920×1080 resolution image, about 2MB per frame, 60MB per second at 30fps). It requires a lot of computing resources to process the static part, which is a waste of computing power.

[0005] Severe motion blur: The exposure time of frame-based cameras (e.g., 1ms) causes motion blur of fast-moving objects, affecting the accuracy of feature point matching and reducing the precision of 3D reconstruction.

[0006] Poor dynamic adaptability: Traditional methods need to process all pixels of the entire image, cannot distinguish between static and dynamic areas, and have low efficiency in perceiving dynamic scenes.

[0007] To address the aforementioned issues, event cameras have emerged. Event cameras employ an asynchronous triggering mechanism, outputting events (including timestamps, pixel coordinates, and polarity (brightness increase / decrease)) only when pixel brightness changes. They offer advantages such as microsecond-level temporal resolution (e.g., 1 μs), data sparsity (e.g., 1 meps, 1 million events per second), and no motion blur. However, existing 3D reconstruction methods using event cameras still suffer from the following shortcomings:

[0008] The indirect processing method of "accumulating events into frames" is often used, which does not fully utilize the asynchronous nature of events, and the latency is still relatively high.

[0009] The matching algorithm does not fully incorporate the spatiotemporal features of the event (timestamp, polarity, spatial location), resulting in low matching accuracy.

[0010] The lack of hardware-level synchronization and real-time matching solutions for binocular event streams makes it impossible to achieve ultra-low latency. Summary of the Invention

[0011] To address the technical problems existing in the prior art, the present invention provides the following technical solution:

[0012] On the one hand, a 3D reconstruction method based on binocular event flow matching is provided, which is implemented by an electronic device, and the method includes:

[0013] S1. Binocular event camera calibration: Calibrate two event cameras to obtain the camera's intrinsic and extrinsic parameters and baseline distance;

[0014] S2. Event stream acquisition and preprocessing: The event stream is acquired asynchronously from the two event cameras, and the event stream is subjected to spatiotemporal joint denoising preprocessing.

[0015] S3. Event Stream Synchronization Verification: Perform synchronization verification on the denoised binocular event stream to eliminate timestamp errors between the left and right camera event streams;

[0016] S4. Event matching based on spatiotemporal constraints: Based on spatiotemporal constraints, the events in the synchronized left camera event stream are matched with the events in the right camera event stream to obtain matching event pairs;

[0017] S5. 3D point cloud generation: Based on the matching event pairs, the depth is calculated using the binocular triangulation principle and converted into three-dimensional coordinates to generate a sparse 3D point cloud.

[0018] S6. Point cloud post-processing: Filtering and time consistency processing are performed on the sparse 3D point cloud to obtain a purified 3D point cloud.

[0019] S7. Point Cloud Output: Outputs the purified 3D point cloud.

[0020] Preferably, the matching based on spatiotemporal constraints in step S4 specifically includes:

[0021] For an event in the left camera event stream, apply the following constraints sequentially to filter candidate matching events from the right camera event stream:

[0022] Time constraint: The difference between the timestamp of the candidate event and the timestamp of the left camera event is within a preset threshold;

[0023] Polarity constraint: The polarity of the candidate event is the same as the polarity of the left camera event;

[0024] Epipolar constraint: The pixel coordinates of the candidate event lie on the epipolar line determined by the left camera event and the camera extrinsic parameters;

[0025] Among the candidate events that satisfy the above constraints, the final matching event is determined by comparing the similarity of the event's temporal surface feature vectors.

[0026] Preferably, the time surface feature is obtained by calculating the gradient of the time surface at the location of the event pixel within a preset time window, and the gradient includes a horizontal gradient Gx and a vertical gradient Gy.

[0027] Preferably, the synchronization verification in step S3 specifically employs cross-correlation analysis to calculate the temporal correlation of the event streams of the left and right cameras, and adjusts the timestamps to maximize the cross-correlation function value, thereby eliminating synchronization errors.

[0028] Preferably, the spatiotemporal joint denoising in step S2 includes:

[0029] Spatial constraint denoising: If an event has no other events within its preset spatial neighborhood and within a preset time window, it is determined to be isolated noise and filtered out;

[0030] Time-constrained noise reduction: If the total number of events within a preset time window before and after an event is less than a threshold, it is determined to be thermal noise and filtered out.

[0031] Preferably, the filtering in step S6 includes statistical filtering and radius filtering, and the time consistency processing is to filter out 3D points that appear only once within a continuous time window.

[0032] Preferably, the calibration in step S1 includes: obtaining the intrinsic parameters of the two event cameras using a checkerboard calibration plate, and obtaining the extrinsic parameters and baseline distance describing the relative positional relationship between the two cameras through stereo calibration.

[0033] Preferably, the total latency of the method is less than 30 microseconds, the data rate of the event stream is up to 1 million events per second, and the refresh rate of the 3D point cloud is consistent with the event rate.

[0034] On the other hand, a 3D reconstruction system based on binocular event flow matching for implementing the above-described method is provided, comprising:

[0035] The binocular event camera module contains two synchronous event cameras for asynchronously acquiring event streams from the scene;

[0036] The synchronization module uses a field-programmable gate array (FPGA) to generate synchronization pulse signals to ensure that the timestamps of the two event cameras are synchronized.

[0037] The processing unit, including a central processing unit (CPU) and a graphics processing unit (GPU), is used to run algorithms for event stream preprocessing, synchronous verification, event matching, 3D point cloud generation, and post-processing.

[0038] The synchronization module is connected to the binocular event camera module, the binocular event camera module is connected to the processing unit, and the processing unit outputs 3D point cloud data.

[0039] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for 3D reconstruction based on binocular event stream matching.

[0040] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described 3D reconstruction methods based on binocular event stream matching.

[0041] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0042] Compared with traditional 3D reconstruction methods and existing event camera methods, this solution has the following core technical advantages:

[0043] (i) Ultra-low latency, adaptable to high-speed dynamic scenarios

[0044] Using an event camera (1μs time resolution) and hardware acceleration (FPGA+GPU), the total system latency is <30μs, which is much lower than that of traditional frame camera methods (≥100ms).

[0045] It supports perception of high-speed dynamic scenes (such as car collisions and robots grasping ping-pong balls), with latency meeting real-time control requirements (<100μs).

[0046] (ii) Sparse data, high computational efficiency

[0047] Event cameras only output events in dynamic regions (such as the edges of moving objects), and the data volume (1 Meps, 4 MB / s) is much smaller than that of frame cameras (30 fps, 60 MB / s).

[0048] The matching algorithm only processes dynamic events and does not need to process static backgrounds, reducing computational resource consumption by ≥90%.

[0049] (iii) Accurate matching and high 3D point cloud precision

[0050] Combining spatiotemporal constraints (timestamp, polarity, epipolarity) and temporal surface features (Gx, Gy), the matching accuracy is ≥95% (false matching rate <5% in dynamic scenarios).

[0051] The depth accuracy of 3D point clouds is ≥98% (depth error <1% in dynamic scenes), which is higher than that of traditional frame camera methods (≥90%).

[0052] (iv) No motion fuzziness, strong dynamic adaptability

[0053] The asynchronous triggering mechanism of the event camera (no exposure time) eliminates motion blur and ensures that the feature points of high-speed moving objects are clear;

[0054] Point clouds contain only points in dynamic regions, adapting to the perception needs of dynamic scenes (such as robot navigation that needs to focus on moving objects).

[0055] (v) Hardware-level synchronization, high reliability

[0056] Hardware-level synchronization of the binocular event camera is implemented using FPGA (time error <1μs) to ensure the temporal consistency of the event stream;

[0057] The system uses industrial-grade hardware (EVK4 event camera, Jetson AGX Orin), supports wide temperature range (-40℃~85℃) and vibration resistance (10G), making it suitable for industrial environments. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of the hardware system composition of a 3D reconstruction system based on binocular event flow matching provided in an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of the software system functional architecture of a 3D reconstruction system based on binocular event flow matching provided in an embodiment of the present invention;

[0061] Figure 3 This is a flowchart of a 3D reconstruction method based on binocular event flow matching provided by an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0063] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0064] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0065] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0066] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0067] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0068] This invention abandons the traditional "frame" concept of cameras, utilizing the high temporal resolution and data sparsity of event cameras to achieve ultra-low latency dynamic scene 3D perception. This method employs two synchronized event cameras. Instead of processing image frames, it directly matches spatiotemporally corresponding "event" streams from the left and right cameras in real time at the hardware level. Successfully matched event pairs directly generate a sparse but extremely high refresh rate 3D point cloud, which only contains the parts of the scene that have changed or moved.

[0069] The implementation plan of this solution will be described in detail below, in conjunction with the introduction of this system.

[0070] This solution adopts an architecture of "binocular event camera + hardware acceleration processing unit + real-time software stack", which fully leverages the asynchronous characteristics of the event camera and the parallel computing capabilities of the hardware to achieve ultra-low latency dynamic scene 3D reconstruction.

[0071] I. Hardware System Composition

[0072] like Figure 1 As shown, the hardware system includes a binocular event camera module, a synchronization module, a processing unit, a storage unit, a power supply module, and an interface module, which are described in detail below:

[0073] 1. Binocular event camera module

[0074] Two Prophesee EVK4 event cameras were selected, with the following main parameters:

[0075] Resolution: 640×480 pixels;

[0076] Time resolution: 1μs (event timestamp precision);

[0077] Event rate: Maximum 1 Meps (1 million events per second);

[0078] Polarity: Supports brightness increase (+1) / decrease (-1) output;

[0079] Interface: USB 3.2 Gen 2 (10Gbps, for transmitting event streams);

[0080] Lens: Focal length f=3mm (fixed focus, 60° angle of view), distortion coefficient <0.1%.

[0081] Description: The EVK4 event camera's high temporal resolution and low distortion characteristics ensure that it can capture details in fast-moving scenes; the USB 3.2 interface meets the high-speed transmission requirements of event streams (1Meps event stream is about 4MB / s, far lower than the 10Gbps bandwidth of USB 3.2).

[0082] 2. Synchronization Module

[0083] A Xilinx Artix-7 FPGA (XC7A35T) was selected to implement hardware-level synchronization of a stereo event camera. Its main functions include:

[0084] A synchronization pulse signal (frequency 100kHz, pulse width 1μs) is generated and sent to the two event cameras via the GPIO interface to ensure their timestamp synchronization.

[0085] The system monitors the event stream timestamps of the two cameras in real time, calculates the synchronization error (such as time difference), and adjusts the internal clock of the cameras via the I2C interface to keep the synchronization error within <1μs.

[0086] Description: The Artix-7 FPGA's parallel processing capability (up to 100k logic units) enables microsecond-level synchronous control, ensuring the time consistency of binocular event streams.

[0087] 3. Processing Unit

[0088] The NVIDIA Jetson AGX Orin (8-core ARM Cortex-A78AE CPU + 2048-core Ampere GPU) is selected as the computing core of the system, with the following main functions:

[0089] Receive and preprocess the stereo event stream;

[0090] Implement algorithms such as event matching and 3D point cloud generation;

[0091] Control the coordinated operation of each module.

[0092] Description: The Jetson AGX Orin GPU (supporting CUDA 11.8) has powerful parallel computing capabilities (peak computing power of 275 TOPS), making it suitable for handling parallel tasks such as event matching; the CPU is used to handle serial tasks such as synchronous verification and point cloud post-processing, realizing heterogeneous computing of "CPU+GPU".

[0093] 4. Storage Unit

[0094] High-speed cache: Samsung DDR4-3200 (16GB) is used to cache real-time event streams (such as caching events within 1 second, about 4MB), ensuring data read latency <100ns;

[0095] Non-volatile storage: Employs a Western Digital NVMe SSD (512GB) for storing camera calibration parameters, algorithm configuration files, etc.

[0096] 5. Power Module

[0097] The Mean Well LRS-200-24 (24V / 8.3A) switching power supply is selected to provide stable power to the binocular event camera, synchronization module, and processing unit, ensuring the reliability of the system in industrial environments.

[0098] 6. Interface Module

[0099] Input interfaces: USB 3.2 Gen 2 (2 ports), used to connect two event cameras and transmit event streams;

[0100] Output interfaces: PCIe 4.0 (1, 16Gbps), for outputting 3D point clouds to display devices (such as 3D monitors); Ethernet (1, 1Gbps), for transmitting point clouds to robot navigation systems;

[0101] Debugging interface: UART (1 unit, 115200bps), used for system debugging and log output.

[0102] II. Software System Composition

[0103] like Figure 2 As shown, the software system adopts a layered architecture of "low-level driver + real-time preprocessing + core algorithm + post-processing + output," implemented in C++ / CUDA to ensure low latency and high reliability. The software stack structure is as follows:

[0104] 1. Low-level driver layer

[0105] Event camera driver: Based on Prophesee Metavision SDK (version 2.20), it implements asynchronous acquisition of event streams (via USB 3.2 interface) and supports event filtering (such as filtering by timestamp range).

[0106] Synchronization module driver: Based on Xilinx Vivado SDK (version 2023.1), it implements the generation of FPGA synchronization pulses and the calibration of event stream timestamps.

[0107] 2. Real-time preprocessing layer

[0108] Event denoising module: Employs a spatiotemporal joint denoising algorithm to remove thermal noise and isolated noise from event cameras. Specific logic:

[0109] Spatial constraint: For each event, check the events in its 3×3 neighborhood (caching the events in the most recent 1ms). If there are no events in the neighborhood, they are considered isolated noise and discarded.

[0110] Time constraint: For each event, count the number of events within 500μs before and after it. If the number is less than 5, it is considered thermal noise and discarded.

[0111] Event stream sorting module: Sorts the binocular event streams in ascending order of timestamps to prepare for subsequent synchronous verification and matching.

[0112] 3. Core Algorithm Layer

[0113] (1) Synchronization verification module: The cross-correlation analysis method is used to calculate the temporal correlation of the binocular event stream and adjust the timestamp to eliminate synchronization error.

[0114] (2) Event matching module: A fast matching algorithm based on spatiotemporal constraints is adopted, which combines the timestamp, polarity, spatial location and features of the event to achieve low-latency matching. See the "Method and Steps" section for the specific logic.

[0115] (3) 3D point cloud generation module: Based on the principle of binocular triangulation, the depth is calculated by matching event pairs to generate 3D point clouds.

[0116] 4. Post-processing layer

[0117] Point cloud filtering module: Statistical filtering (removing outliers) and radius filtering (removing isolated points) are used, implemented based on the PCL library (version 1.12);

[0118] Time consistency processing module: Tracks point clouds within a continuous time window (e.g., 10ms), removes noise points that appear only once, and improves the temporal continuity of the point cloud.

[0119] 5. Output Layer

[0120] Point cloud display module: Based on OpenGL library (version 4.6), it realizes real-time rendering of 3D point clouds (frame rate ≥ 1000fps).

[0121] Data transmission module: Based on the Ethernet / IP protocol, it transmits point cloud data (format: PCD) to the robot navigation system (latency <1ms).

[0122] III. System Data Communication and Interaction

[0123] The system's data flow and interaction process are as follows (taking the scenario of "robot grasping fast-moving objects" as an example):

[0124] (a) Data communication between hardware

[0125] Synchronization signal transmission: The FPGA synchronization module generates a synchronization pulse (100kHz, 1μs width) and sends it to the two EVK4 event cameras through the GPIO interface to ensure that their internal clocks are synchronized (time error <1μs).

[0126] Event stream transmission: Two EVK4 event cameras transmit asynchronous event streams (each event occupies 4 bytes: 2 bytes timestamp + 2 bytes coordinate + 1 bit polarity) to the Jetson AGX Orin USB controller via a USB 3.2 Gen 2 interface (transmission latency <10μs).

[0127] Processing result transmission: Jetson AGX Orin transmits the post-processed 3D point cloud (format: PCD) to the 3D display via PCIe 4.0 interface (transmission latency <5μs), or to the robot controller via Ethernet (transmission latency <1ms).

[0128] (II) Inter-software interaction process (detailed description of each step is provided in Part IV below)

[0129] Event stream acquisition: The Metavision SDK driver reads the event stream from the USB controller and stores it in the DDR4 cache (latency <100ns).

[0130] Preprocessing: The event denoising module (running on the FPGA) reads the event stream from DDR4, performs spatiotemporal joint denoising (delay <1μs), and returns the denoised event stream to DDR4.

[0131] Synchronization verification: The synchronization verification module (running on the CPU) reads the denoised stereo event stream from DDR4, performs cross-correlation analysis (delay <0.5μs), and adjusts the timestamp of the right camera event stream (to eliminate synchronization errors).

[0132] Event matching: The event matching module (running on the GPU) reads the synchronized stereo event stream from DDR4, executes a matching algorithm based on spatiotemporal constraints (latency < 5μs), and outputs the matched event pairs.

[0133] 3D point cloud generation: The 3D point cloud generation module (running on the GPU) reads the matching event pairs from the GPU memory, performs triangulation (latency <1μs), and generates a 3D point cloud (stored in GPU memory).

[0134] Post-processing: The point cloud filtering module (running on the CPU) reads the point cloud from the GPU memory, performs statistical filtering and radius filtering (delay <0.5μs), and outputs a clean point cloud.

[0135] Output: The point cloud display module (running on the CPU) reads the point cloud from the CPU memory, performs OpenGL rendering (latency <1μs), and displays the 3D point cloud on the monitor; at the same time, the data transmission module transmits the point cloud to the robot controller (latency <1ms).

[0136] (III) System Delay Analysis

[0137] The total system latency (from event generation to point cloud output) is calculated as follows:

[0138] Event camera delay: 10μs (event output delay of EVK4);

[0139] Event transmission latency: 10μs (latency of transmitting a 1Meps event stream via USB 3.2);

[0140] Preprocessing delay: 1μs (FPGA denoising);

[0141] Synchronization verification latency: 0.5μs (CPU cross-correlation analysis);

[0142] Event matching latency: 5μs (GPU matching);

[0143] 3D point cloud generation latency: 1μs (GPU triangulation);

[0144] Post-processing delay: 0.5μs (CPU filtering);

[0145] Output delay: 1μs (PCIe transmission to display).

[0146] Total latency: 10+10+1+0.5+5+1+0.5+1=29μs, which meets the latency requirements for robot navigation (<100μs) and high-speed grasping (<50μs).

[0147] IV. Detailed introduction of methods and steps

[0148] like Figure 3 As shown, the core of this solution is "real-time matching of binocular event streams and 3D point cloud generation". The method steps include: camera calibration, event stream acquisition and preprocessing, synchronous verification, event matching, 3D point cloud generation, post-processing, and output. The implementation process, technical principles, and technical effects of each step are as follows:

[0149] (I) Step 1: Binocular event camera calibration

[0150] Implementation process:

[0151] Intrinsic parameter calibration: Using a checkerboard pattern (10×10 grid, 20mm grid size), two EVK4 event cameras were mounted on a tripod, and 20 checkerboard images from different angles were captured (events were accumulated into frames, frame rate 1fps). Intrinsic parameters were calculated using OpenCV's `calibrateCamera` function.

[0152] Focal length: f_x = 600 pixels, f_y = 600 pixels;

[0153] Principal points: c_x = 320 pixels, c_y = 240 pixels;

[0154] Distortion coefficients: k1=0.01, k2=-0.005, p1=0.001, p2=0.0005.

[0155] Extrinsic parameter calibration: Place two cameras horizontally (baseline distance B = 0.5 meters) and take 20 stereo checkerboard images. Calculate the extrinsic parameters using OpenCV's stereoCalibrate function.

[0156] Rotation matrix: R=[[1,0,0],[0,1,0],[0,0,1]] (camera optical axis parallel);

[0157] Translation vector: T = [-0.5, 0, 0] (baseline direction is the X-axis).

[0158] Parameter storage: Store the intrinsic parameters (f_x,f_y,c_x,c_y,k1,k2,p1,p2) and extrinsic parameters (R,T,B) in the NVMeSSD configuration file (format: YAML).

[0159] Intrinsic calibration uses the known dimensions of the checkerboard to calculate the camera's focal length, principal point, and distortion coefficient, eliminating the impact of lens distortion on the image. Extrinsic calibration uses corresponding points on the 3D checkerboard to calculate the relative position (rotation and translation) of the two cameras, providing a basis for subsequent epipolar constraints.

[0160] After internal parameter calibration, the lens distortion error is <0.5 pixels; after external parameter calibration, the baseline distance measurement error is <1mm, ensuring the depth accuracy of the 3D point cloud (<1%).

[0161] (ii) Step 2: Event Stream Acquisition and Preprocessing

[0162] Implementation process:

[0163] Event stream acquisition: The event stream is read asynchronously from the USB interface using the EventStream class in the Metavision SDK. The structure of each event is as follows:

[0164] struct Event {

[0165] uint32_t timestamp; / / Timestamp (μs)

[0166] uint16_t x; / / Pixel x-coordinate (0~639)

[0167] uint16_t y; / / Pixel y coordinate (0~479)

[0168] bool polarity; / / Polarity (true: brightness increases, false: brightness decreases)

[0169] }

[0170] Event denoising: A spatiotemporal joint denoising algorithm is used, running on an FPGA (implemented in Verilog).

[0171] Spatial constraint: For each event, check the events in its 3×3 neighborhood (caching the events in the most recent 1ms). If there are no events in the neighborhood, they are considered isolated noise and discarded.

[0172] Time constraint: For each event, count the number of events within 500μs before and after it. If the number is less than 5, it is considered thermal noise and discarded.

[0173] Noise from event cameras (such as thermal noise and isolated noise) often does not satisfy spatial (no events in the neighborhood) or temporal (no events before or after) correlation. Spatiotemporal constraints can effectively filter noise. The noise removal rate is ≥95% (false positive rate of isolated noise and thermal noise <1%). Events in dynamic regions (such as the edges of moving objects) are preserved to ensure the accuracy of subsequent matching.

[0174] (III) Step 3: Event Stream Synchronization Verification

[0175] Implementation process:

[0176] Event stream sorting: Sort the denoised event streams of the left and right cameras in ascending order of timestamps to obtain sequences T1={t1,t2, ..., tn} (left camera) and T2={s1, s2, ..., sm} (right camera).

[0177] 2. Cross-correlation analysis: Specific steps:

[0178] Calculate the cross-correlation function between T1 and T2:

[0179] R(τ) = E[T1(t)·T2(t+τ)],

[0180] in:

[0181] R(τ) is the cross-correlation function value, which measures the correlation strength between the event streams of the left and right cameras at a time difference τ.

[0182] E[·] represents the expectation operator (meaning to take the statistical average of the products within the parentheses (i.e., the sum of the products of all event pairs divided by the number of event pairs)).

[0183] τ is a time difference variable, and its value range is usually the possible range of synchronization error (such as −10μs∼+10μs in the document).

[0184] The cross-correlation function calculates the average product of the left and right event streams at different time differences τ, finding the τ0 with the largest R(τ). This τ0 represents the synchronization error. Because the timestamp sequences of the synchronized event streams have the highest correlation (largest average product) at τ=τ0. Adjusting the timestamp of the right camera event (si - τ0) eliminates the synchronization error, ensuring the temporal consistency of the stereo event streams.

[0185] The timestamp sequences of synchronized stereo event streams should have high correlation, with the maximum value of the cross-correlation function corresponding to the synchronization error τ0. The synchronization error was further reduced from <1μs after calibration to <2μs (including transmission delay); this ensures that subsequent matched event pairs come from the same time point, improving the accuracy of the 3D point cloud.

[0186] (iv) Step 4: Event matching based on spatiotemporal constraints (core step)

[0187] Implementation process:

[0188] For each event e1=(x1, y1, t1, p1) of the left camera (x1, y1: pixel coordinates; t1: timestamp; p1: polarity), perform the following steps:

[0189] Epipolar constraint calculation: Based on the camera extrinsic parameters (rotation matrix R, translation vector T), calculate the epipolar equation of e1:

[0190] ,

[0191] Where (x2,y2) are the pixel coordinates of the right camera, and a, b, and c are epipolar parameters (derived from intrinsic and extrinsic parameters).

[0192] Time-constrained filtering: Extract events with timestamps within [t1-Δt, t1+Δt] (Δt=2μs) from the right camera event stream to obtain set E2 (Δt is the synchronization error threshold).

[0193] Polarity constraint filtering: Filter out events with polarity p2=p1 from E2 (the same object's motion has the same polarity in both cameras) to obtain set E2'.

[0194] Epipolar constraint filtering: Filter out events located on the epipolar line e1 (error < 1 pixel) from E2' to obtain set E2''.

[0195] Feature extraction: For each event e2=(x2,y2,t2,p2) in e1 and E2'', calculate the time surface feature:

[0196] Time surface: T(x,y,t) represents the last event timestamp of pixel (x,y) at time t;

[0197] Gradient calculation: The Sobel operator (which captures the direction and intensity of brightness changes by weighting a 3×3 neighborhood of pixels on the time surface) is used to calculate the gradient (Gx, Gy) of the time surface, which serves as the feature vector of the event.

[0198] ,

[0199] ,

[0200] Formula Explanation: Gx and Gy are the horizontal and vertical gradients of the time surface, respectively, calculated using the Sobel operator convolution. Gx is positive on the right and negative on the left (for detecting horizontal edges; the right pixel weights are positive and the left weights are negative, used to detect horizontal brightness changes (such as the vertical edges of an object)). Gy is positive at the bottom and negative at the top (for detecting vertical edges; the bottom pixel weights are positive and the top weights are negative, used to detect vertical brightness changes (such as the horizontal edges of an object)).

[0201] Gradients capture the spatial features of events, providing a basis for subsequent matching. A larger gradient value indicates a more drastic change in brightness at that pixel location, representing a key feature of object edges in dynamic scenes. These gradient values ​​form the event's feature vector, used for subsequent event matching.

[0202] Feature matching: Calculate the Euclidean distance between the feature vectors (Gx1, Gy1) of e1 and (Gx2, Gy2) of e2.

[0203] ,

[0204] Formula explanation: d is the Euclidean distance between the feature vectors of the two events. / It is the left event feature (that is, the temporal surface gradient features (horizontal and vertical directions) of the left camera event e1). / It is the right event feature (that is, the temporal surface gradient features (horizontal and vertical directions) of the right camera event e2).

[0205] The smaller the Euclidean distance, the more similar the feature vectors of the two events are. In event matching, for left camera event e1, candidate events e2 that satisfy the time, polarity, and epipolarity constraints are selected from right camera events, and their Euclidean distance d is calculated. e2, with the smallest d and less than a threshold (e.g., 0.1), is selected as the matching event for e1 to ensure matching accuracy.

[0206] Time constraint (Δt=2μs): Ensures that the matched event pairs come from the same point in time, eliminating motion errors in dynamic scenes;

[0207] Polarity constraint (p1=p2): The brightness change direction of the same object's motion is consistent in both cameras, filtering out incorrect matches;

[0208] Epipolar constraint (e2 lies on the epipolar line of e1): A fundamental constraint of binocular vision, requiring that the two image points of the same spatial point lie on the epipolar line, thus reducing the matching range;

[0209] Temporal surface features (Gx, Gy): Spatiotemporal characteristics of events, reflecting the direction and speed of pixel brightness changes, improving matching accuracy. Furthermore, the matching latency is <5μs (GPU parallel processing, processing 1 Meps of event stream per second).

[0210] (V) Step 5: 3D point cloud generation

[0211] Implementation process:

[0212] For each matching event pair (e1, e2), perform the following steps:

[0213] Parallax calculation: Assuming the camera is placed horizontally (the baseline direction is the X-axis), the parallax is d = x1 - x2 (x1 and x2 are the pixel x-coordinates of the left and right cameras).

[0214] Depth calculation: Calculate depth Z0 (unit: meters) using binocular triangulation formulas:

[0215]

[0216] Where f = 600 pixels (focal length, intrinsic parameter) and B = 0.5 meters (baseline distance, extrinsic parameter).

[0217] Formula explanation: In depth calculation, Z0 is the spatial point depth (i.e., the vertical distance from the point to the camera plane, in meters), f is the camera focal length (pixels), B is the binocular baseline distance (the horizontal distance between the optical centers of the two cameras, in meters), and d is the parallax (the difference between the left camera pixel x1 and the corresponding right camera pixel x2, x1-x2).

[0218] Based on the principle of similar triangles, parallax is inversely proportional to depth, enabling accurate depth calculation in dynamic scenes. The optical centers of two cameras and a spatial point form similar triangles, with baseline B as the base of the triangle and focal length f as the distance from the camera plane to the optical center. Parallax d is inversely proportional to depth Z0: the larger the parallax, the smaller the depth (the closer the object is to the camera); the smaller the parallax, the larger the depth (the farther the object is from the camera). The high temporal resolution of the event camera ensures no motion blur in the parallax calculation, resulting in extremely high depth accuracy.

[0219] 3D coordinate transformation: Convert the pixel coordinates (x1, y1) of the left camera to 3D coordinates (X, Y, Z) in the camera coordinate system:

[0220] , Z = Z,

[0221] in, =320 pixels, =240 pixels (master point, intrinsic parameters).

[0222] Formula explanation: In the 3D coordinate transformation formula; , : Pixel coordinates of the left camera event (unit: pixels);

[0223] , : Principal point coordinates of the camera (pixel coordinates of the image center, reference) =320、 =240);

[0224] , The focal length of the camera in the x and y directions (in pixels, e.g.) =600、 =600).

[0225] This formula converts pixel coordinates to 3D world coordinates:

[0226] Eliminate principal point offset: and Convert the pixel coordinates from the top left corner (0,0) of the image to coordinates with the center of the image as the origin;

[0227] Depth scaling: Multiply by depth Z0 to map pixel coordinates to 3D space;

[0228] Unit conversion: Divide by focal length The pixel units are converted to actual meters (because focal length is in pixels and depth is in meters); finally, the 3D coordinates of the spatial point in the camera coordinate system are obtained, which are used to generate point clouds.

[0229] Therefore, by eliminating principal point offset and mapping pixel coordinates to the camera coordinate system through depth scaling, the 2D to 3D conversion is completed.

[0230] Point cloud generation: Store all 3D points (X,Y,Z) in PCD format (point cloud library format), including timestamp (t1) and polarity (p1).

[0231] Binocular triangulation calculates depth using the parallax (pixel difference at the same point in space) between two cameras, a classic method for stereo vision. The high temporal resolution of the event camera ensures accurate parallax calculation (no motion blur). This allows the point cloud refresh rate to equal the event rate (1 Meps), meaning one million 3D points are generated per second, reflecting real-time changes in dynamic scenes.

[0232] (vi) Step 6: Point cloud post-processing

[0233] Implementation process:

[0234] Statistical filtering: For each 3D point, calculate the average distance of its 50 nearest neighbors. If the distance of the point is greater than twice the standard deviation of the average distance, it is considered an outlier and discarded.

[0235] Radius filtering: For each 3D point, count the number of points within its 0.1-meter radius. If the number of points is less than 5, it is considered an isolated point and discarded.

[0236] Time consistency processing: For continuous time windows (e.g., 10ms), track the position changes of each 3D point. If a point appears only once, it is considered a noise point and discarded.

[0237] Statistical filtering: When the distance to outliers is much greater than the average distance to their neighbors, the standard deviation can be used to effectively filter them.

[0238] Radius filtering: When there are no other points in the neighborhood of an isolated point, it can be filtered out by radius constraints;

[0239] Time consistency processing: True dynamic points (such as the edges of moving objects) should exist within a continuous time window; points that appear only once are mostly noise.

[0240] Point cloud noise removal rate ≥90% (false positive rate of outliers and isolated points <1%); preserves continuous point clouds of dynamic objects (such as the outline of a moving car) to ensure the reliability of subsequent applications (such as robot navigation).

[0241] (vii) Step 7: Point cloud output

[0242] Implementation process:

[0243] Point cloud display: The post-processed point cloud (PCD format) is transmitted to a 3D display and rendered using OpenGL (frame rate ≥ 1000fps) to display the 3D structure of dynamic scenes (such as the position and speed of moving objects).

[0244] Data transmission: The point cloud is transmitted to the robot controller (such as the ROS system) via Ethernet for navigation (obstacle avoidance) or grasping (path planning).

[0245] OpenGL hardware-accelerated rendering (GPU) ensures real-time display of point clouds; the high bandwidth (1Gbps) of the Ethernet / IP protocol ensures fast transmission of point clouds. Point cloud display latency is <1μs (PCIe transmission); data transmission latency is <1ms (Ethernet), meeting the requirements of real-time robot control.

[0246] Therefore, this solution, through an architecture of "binocular event camera + hardware acceleration + real-time algorithm," fully leverages the asynchronous characteristics of the event camera and the parallel computing capabilities of the hardware to achieve ultra-low latency (<30μs), high accuracy (matching accuracy ≥95%), and high sparsity (data volume reduction ≥90%) in dynamic scene 3D reconstruction. Compared to traditional methods, this solution has significant advantages in high-speed dynamic scenes (such as robot grasping and autonomous driving) and can be widely applied in fields such as industrial robots, autonomous driving, and virtual reality.

[0247] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, electronic device 410 may include a first processor 2001.

[0248] Optionally, the electronic device 410 may also include a memory 2002 and a transceiver 2003.

[0249] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0250] The following is combined with Figure 4 A detailed description of each component of electronic device 410 is provided below:

[0251] The first processor 2001 is the control center of the electronic device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0252] Optionally, the first processor 2001 can perform various functions of the electronic device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0253] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0254] In a specific implementation, as one example, the electronic device 410 may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0255] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0256] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be connected via the interface circuit of the electronic device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0257] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0258] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0259] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected via the interface circuit of the electronic device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0260] It should be noted that, Figure 4 The structure of the electronic device 410 shown does not constitute a limitation on the router. Actual knowledge structure identification devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0261] Furthermore, the technical effects of the electronic device 410 can be referred to the technical effects of the 3D reconstruction method based on binocular event flow matching described in the above method embodiments, and will not be repeated here.

[0262] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0263] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0264] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0265] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0266] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0267] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0268] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0269] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, systems, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0270] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0271] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0272] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0273] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0274] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A 3D reconstruction method based on binocular event flow matching, characterized in that, The method includes: S1. Binocular event camera calibration: Calibrate two event cameras to obtain the camera's intrinsic and extrinsic parameters and baseline distance; S2. Event stream acquisition and preprocessing: The event stream is acquired asynchronously from the two event cameras, and the event stream is subjected to spatiotemporal joint denoising preprocessing. S3. Event Stream Synchronization Verification: Perform synchronization verification on the denoised binocular event stream to eliminate timestamp errors between the left and right camera event streams; S4. Event matching based on spatiotemporal constraints: Based on spatiotemporal constraints, the events in the synchronized left camera event stream are matched with the events in the right camera event stream to obtain matching event pairs; S5. 3D point cloud generation: Based on the matching event pairs, the depth is calculated using the binocular triangulation principle and converted into three-dimensional coordinates to generate a sparse 3D point cloud. S6. Point cloud post-processing: Filtering and time consistency processing are performed on the sparse 3D point cloud to obtain a purified 3D point cloud. S7. Point Cloud Output: Outputs the purified 3D point cloud.

2. The method according to claim 1, characterized in that, The matching based on spatiotemporal constraints described in step S4 specifically includes: For an event in the left camera event stream, apply the following constraints sequentially to filter candidate matching events from the right camera event stream: Time constraint: The difference between the timestamp of the candidate event and the timestamp of the left camera event is within a preset threshold; Polarity constraint: The polarity of the candidate event is the same as the polarity of the left camera event; Epipolar constraint: The pixel coordinates of the candidate event lie on the epipolar line determined by the left camera event and the camera extrinsic parameters; Among the candidate events that satisfy the above constraints, the final matching event is determined by comparing the similarity of the event's temporal surface feature vectors.

3. The method according to claim 2, characterized in that, The time surface features are obtained by calculating the gradient of the time surface at the location of the event pixel within a preset time window. The gradient includes the horizontal gradient Gx and the vertical gradient Gy.

4. The method according to claim 1, characterized in that, The synchronization verification described in step S3 specifically uses cross-correlation analysis to calculate the temporal correlation of the event streams of the left and right cameras, and adjusts the timestamps to maximize the cross-correlation function value, thereby eliminating synchronization errors.

5. The method according to claim 1, characterized in that, The spatiotemporal joint denoising described in step S2 includes: Spatial constraint denoising: If an event has no other events within its preset spatial neighborhood and within a preset time window, it is determined to be isolated noise and filtered out; Time-constrained noise reduction: If the total number of events within a preset time window before and after an event is less than a threshold, it is determined to be thermal noise and filtered out.

6. The method according to claim 1, characterized in that, The filtering in step S6 includes statistical filtering and radius filtering, and the time consistency processing is to filter out 3D points that appear only once within a continuous time window.

7. The method according to claim 1, characterized in that, The calibration in step S1 includes: using a checkerboard calibration board to obtain the intrinsic parameters of the two event cameras respectively, and obtaining the extrinsic parameters and baseline distance describing the relative positional relationship between the two cameras through stereo calibration.

8. The method according to any one of claims 1, characterized in that, The total latency of the method is less than 30 microseconds, the data rate of the event stream is up to 1 million events per second, and the refresh rate of the 3D point cloud is consistent with the event rate.

9. A 3D reconstruction system based on binocular event flow matching for implementing the method of any one of claims 1-8, characterized in that, include: The binocular event camera module contains two synchronous event cameras for asynchronously acquiring event streams from the scene; The synchronization module uses a field-programmable gate array (FPGA) to generate synchronization pulse signals to ensure that the timestamps of the two event cameras are synchronized. The processing unit, including a central processing unit (CPU) and a graphics processing unit (GPU), is used to run algorithms for event stream preprocessing, synchronous verification, event matching, 3D point cloud generation, and post-processing. The synchronization module is connected to the binocular event camera module, the binocular event camera module is connected to the processing unit, and the processing unit outputs 3D point cloud data.

Images