Three-dimensional motion estimation method based on event camera

Abstract

The invention discloses a three-dimensional motion estimation method based on an event camera. The method comprises the following steps: 1, constructing a time-space decoupling event time sequence and spatial representation; 2, geometric structure and context features are extracted through a space encoder, and time sequence evolution features are extracted through a time sequence encoder; 3, respectively constructing a spatial correlation graph and a time sequence correlation graph based on the geometric structure characteristics and the time sequence evolution characteristics; 4, constructing a pixel track curve according to the event density; 5, geometric structure and time sequence evolution characteristics are sampled along the pixel track curve, and a space-time correlation graph is generated; 6, iteratively optimizing pixel track curve parameters according to the space-time correlation graph and the context features; and 7, utilizing the optimized pixel track curve and the derivative thereof to accurately predict the three-dimensional motion. The method obviously improves the three-dimensional motion estimation precision of the event camera in a complex scene, is suitable for the fields of automatic driving and robots, and has a good engineering application value.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically a three-dimensional motion estimation method based on an event camera. Background Technology

[0002] 3D motion estimation aims to recover the dynamic structural information of a scene from visual data and is widely used in fields such as robot navigation, autonomous driving, and augmented reality. Traditional methods mostly rely on consecutive frame images, estimating the 3D motion of objects or cameras by matching feature points or dense pixels between consecutive frames and combining multi-view geometric models. Although stable in static or slowly changing environments, local smoothness constraints easily fail under complex conditions with rapid motion, abrupt depth changes, or occlusion interference, leading to a significant decrease in estimation accuracy. Event cameras, as a novel type of visual sensor, possess microsecond-level temporal resolution and extremely high dynamic response capabilities, providing more granular spatiotemporal information support for 3D motion modeling. Compared to traditional frame images, event data can continuously capture the process of scene brightness changes, significantly reducing latency while maintaining a high dynamic range, making it suitable for fine motion analysis under high-speed and complex lighting conditions. However, research on event-based 3D motion estimation is still in its early stages. Existing methods mostly focus on local optical flow estimation or dense depth reconstruction within a short period of time, making it difficult to effectively model long-term temporal dependencies or accurately recover point-level depth motion information. Especially in regions with drastic depth changes, the traditional smoothness assumption fails, and how to fully utilize the temporal continuity and sparse activation characteristics of events to achieve continuous modeling of scene motion and motion depth estimation remains a significant challenge for current research. Therefore, constructing an event-driven 3D motion estimation framework with fine-grained temporal modeling capabilities holds promise for overcoming the limitations of existing methods and promoting the in-depth development of event vision in dynamic scene understanding. Summary of the Invention

[0003] This invention addresses the shortcomings of existing event camera-based 3D motion estimation methods in handling scenes with varying depths, such as insufficient spatial representation accuracy and inadequate temporal modeling capabilities. It proposes a 3D motion estimation method based on event cameras to explicitly model the continuous motion trajectory of the scene, thereby improving its adaptability to complex scenes.

[0004] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The present invention provides a three-dimensional motion estimation method based on an event camera, characterized by the following steps: Step 1: Obtain the event sequence and its corresponding pixel-level optical flow tag sequence Y O and motion depth label Y D Thus, the event space representation V and the event sequence representation K are constructed. x Event sequence representation Kyy ; Step 2: Construct a spatial encoder based on a convolutional neural network to extract the geometric structural features F of the event space representation V. hw and contextual features F c ; Step 3: Construct a temporal encoder based on one-dimensional convolution, and start from K x and K y Extracting lateral temporal features F from each wt and longitudinal time series features F ht ; Step 4, based on K x and K y and F wt and F ht Construct spatial correlation diagram Time-series correlation diagram ; Step 5: Extract the top n time index sequences t with the highest density from the spatiotemporal density vector D along the time axis. l The gradient at the corresponding position is calculated to obtain the initial control point P, thereby obtaining the pixel trajectory curve. ; Step 6: On the pixel trajectory curve Guided by this, spatial correlation maps Time-series correlation diagram Sampling is performed to construct the spatiotemporal correlation graph C. st ; Step 7: Use equation (11) to obtain the spatiotemporal feature F st ; F st = Concat(C st , F c (11) In equation (11), Concat( ) represents the concatenation function; Step 8: Calculate the updated control point vector P using equation (12). next ; P next =P + GRU(F st , P)(12) In equation (12), GRU( () represents a gated loop unit; + indicates residual operation; Step 9, P next As P, the control point vector is updated again according to the process of steps 5-8 until the maximum number of updates is reached, thus obtaining the final control point vector P. final Determine the pixel trajectory curve The shape; Step 10: Analyze the pixel trajectory curve. Sampling is performed to obtain the scene optical flow prediction value O. pred and scene motion depth prediction value D pred ; Step 11, based on O pred and Y O and D pred and A loss function L is constructed, and the AdamW optimizer is used to train a 3D motion estimation network consisting of a spatial encoder, a temporal encoder, and a gated recurrent unit. Training stops when the loss L is minimized, thus obtaining a 3D motion estimation model with optimal parameters, which is used to process input event sequences. Perform three-dimensional motion estimation to predict optical flow and motion depth.

[0005] The characteristic of the three-dimensional motion estimation method based on event camera described in this invention is that, in step 1, the event space representation V and the event temporal representation K are constructed using equations (1) and (2) respectively. x Event sequence representation K y ; (1) (2) In equations (1) and (2), i represents the event number; p i x i y i , t i Represent The polarity, x-axis coordinate, y-axis coordinate, and timestamp of the i-th event; k( (x,y) represents the triangular sampling kernel function; (x,y) represents the spatial coordinates of V, and b represents the temporal coordinates of V. These are parameters used to control the smoothness of the Gaussian kernel function.

[0006] Furthermore, in step 4, spatial correlation diagrams are constructed using equations (3) to (7). Time-series correlation diagram : (3) (4) (5) (6) (7) In equations (3)-(7), E represents the Hartmann product; E represents the event sequence. The distribution vector; ( () represents a three-dimensional average pooling operation; D represents an event sequence. The spatiotemporal density vector; ( ) represents a two-dimensional average pooling operation; ( ) represents a one-dimensional average pooling operation; C wt C ht These represent time series correlation diagrams along the time series and horizontal axis, and along the time series and vertical axis, respectively. Represents the outer product of vectors; Furthermore, in step 11, the loss L is calculated using equations (13)-(16): (13) (14) (15) L = + + (16) In equations (13)-(16), | | is the absolute value operation; This refers to the loss in the optical flow component; This represents the loss in the depth of motion. For time-constrained loss; ( () represents the pixel trajectory curve The derivative of T; T is the sequence of events. Length, represent The timestamp of the (i+1)th event.

[0007] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program supporting the processor in executing the three-dimensional motion estimation method, and the processor is configured to execute the program stored in the memory.

[0008] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, performs the steps of the three-dimensional motion estimation method.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention designs an event encoding method that, taking into account the characteristic that event data reflects continuous time motion, uses a continuous time projection kernel to explicitly encode fine-grained time evolution, thereby improving the prediction capability of the three-dimensional motion estimation model for continuous motion.

[0010] 2. This invention proposes a density-based adaptive mechanism that, in response to the motion sensitivity of event data, guides non-uniform spatiotemporal fusion and curve parameter updates, thereby improving the ability of the 3D motion estimation model to perceive depth changes. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the event encoding method designed for this invention; Figure 2 This is a schematic diagram of the density-based adaptive mechanism proposed in this invention; Figure 3 This is a schematic diagram of the 3D motion estimation model framework based on an event camera proposed in this invention. Detailed Implementation

[0012] In this embodiment, considering both the high temporal resolution of event data and the temporal dependence of motion in 3D motion estimation, a 3D motion estimation method based on an event camera is proposed. This method utilizes a continuous-time projection kernel and a density adaptive mechanism to provide an effective solution for 3D motion estimation tasks under high-precision and high-dynamic conditions. Specifically, the 3D motion estimation method proceeds as follows: Step 1: Obtain the event sequence and its corresponding pixel-level optical flow tag sequence Y O and motion depth label Y D Because the raw event data is characterized by non-uniform sampling, asynchronous triggering, and high sparsity, it is difficult to directly input it into a conventional neural network for feature learning. Therefore, as... Figure 1 As shown, the event space representation V and the event timing representation K are constructed using equations (1) and (2), respectively. x Event sequence representation K y ; (1) (2) In equations (1) and (2), i represents the event number; p i x i y i , t i Represent The polarity, x-axis coordinate, y-axis coordinate, and timestamp of the i-th event; k( ) is the triangular sampling kernel function, used to ensure smooth continuity in the spatial and temporal dimensions; (x,y) are the spatial coordinates of V, and b is the temporal coordinate of V; The parameter used to control the smoothness of the Gaussian kernel function is set to 1.0; Step 2: Construct a spatial encoder based on a convolutional neural network to extract the geometric structural features F of the event space representation V. hw and contextual features F c ; Step 3: Construct a temporal encoder based on one-dimensional convolution, and start from K x and K y Extracting the lateral temporal features F respectively wt and longitudinal time series features F ht This design is used to explicitly characterize the continuous motion trend of events in the time dimension, avoiding the problem that traditional two-dimensional convolution is insufficient in modeling temporal sequence information. The temporal convolution kernel size is generally selected as 3 to balance temporal resolution and motion smoothness; Step 4: Construct a spatial correlation diagram using equations (3)-(7). Time-series correlation diagram : (3) (4) (5) (6) (7) In equations (3)-(7), E represents the Hartmann product; E represents the event sequence. The distribution vector; ( () represents a three-dimensional average pooling operation; D represents an event sequence. The spatiotemporal density vector is used to reflect the activity level of events at different spatiotemporal locations, thereby suppressing noisy regions; ( ) represents a two-dimensional average pooling operation; ( ) represents a one-dimensional average pooling operation; C wt C ht These represent time series correlation diagrams along the time series and horizontal axis, and along the time series and vertical axis, respectively. The outer product of vectors is represented by two-dimensional average pooling and one-dimensional average pooling, which are used to eliminate local redundancy in the spatial and planar temporal dimensions, respectively. Finally, a correlation graph that can reflect the consistency of spatial geometry and temporal motion is constructed, providing reliable constraints for subsequent trajectory modeling.

[0013] Step 5: Extract the top n time index sequences t with the highest density from the spatiotemporal density vector D along the time axis. l The gradient at the corresponding location is calculated to obtain the initial control point P, and then the pixel trajectory curve is obtained. The purpose of this step is to automatically locate the main motion trajectory using high-density areas of events, avoiding the instability of manually set initial points. Setting it to 6 is used to strike a balance between accuracy and computational efficiency.

[0014] Step 6: On the pixel trajectory curve Guided by this, spatial correlation maps Time-series correlation diagram Sampling is performed to construct the spatiotemporal correlation graph C. st This design enables the feature extraction process to focus on key regions of real-world moving targets under the guidance of high temporal resolution of events, effectively reducing the interference of background events and noise on 3D motion estimation; Step 7: In order to fuse the low-level motion constraint information with the high-level semantic context information, the spatiotemporal feature F is obtained using equation (11). st ; F st = Concat(C st , F c (11) In equation (11), Concat( ) represents the concatenation function.

[0015] Step 8, as follows Figure 2 As shown, the updated control point vector P is calculated using equation (12). next ; P next =P + GRU(F st , P)(12) In equation (12), GRU( ) represents a gated loop unit, used to model the dynamic relationship of control points over time; + represents residual operation, and the residual structure is used to prevent gradient vanishing and accelerate convergence; Step 9, P next As P, iterative updates are performed multiple times according to steps 5-8 until the maximum number of updates is reached, thereby obtaining the final control point vector P. final And used to determine pixel trajectory curves The shape.

[0016] Step 10: Analyze the pixel trajectory curve. Uniform sampling is performed to obtain the scene optical flow prediction value O. pred and scene motion depth prediction value Dpred This allows for the joint estimation of three-dimensional motion; Step 11: Calculate the loss L using equations (13)-(16), and use the AdamW optimizer to optimize the system consisting of a spatial encoder, a temporal encoder, and a gated loop unit. Figure 3 The 3D motion estimation network shown is trained until the loss L is minimized, at which point training stops, thus obtaining a 3D motion estimation model with optimal parameters, which is then used to process the input event sequence. Perform three-dimensional motion estimation to predict optical flow and motion depth; (13) (14) (15) L = + + (16) In equations (13)-(16), | | is the absolute value operation; This refers to the loss in the optical flow component; This represents the loss in the depth of motion. For time-constrained loss; ( () represents the pixel trajectory curve The derivative of T; T is the sequence of events. Length, Represent The timestamp of the (i+1)th event.

[0017] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0018] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0019] Example: To verify the effectiveness of the method of this invention, this embodiment was tested on the CarlaEvent3D dataset, which contains 15 different sequences and 4425 test samples. The test includes two tasks: optical flow and motion depth estimation. For the optical flow estimation task, two test metrics are used: EPE and f1. EPE measures the average pixel error between the predicted optical flow and the true optical flow. f1 measures the proportion of pixels with an EPE exceeding three pixels. For the motion depth estimation task, log-mid is used as the test metric, which measures the average error of the predicted depth compared to the true value on the order of magnitude.

[0020] In this embodiment, three methods were selected for comparison with the method of the present invention: ScaleFlow, Expansion, and ETTCM. ScaleFlow is derived from the paper "Scaleflow: Efficient deep vision pipeline with closed-loop scale-adaptive inference" published at the 2023 International Multimedia Conference; Expansion is derived from the paper "Learning optical expansion from scale matching" published at the 2023 International Conference on Computer Vision and Pattern Recognition; and ETTCM is derived from the paper "Time-to-contact map by joint estimation of up-to-scale inverse depth and global motion using a single event camera" published at the 2023 International Conference on Computer Vision. The experimental results are shown in Table 1. Table 1. Comparison of results between the method of this invention and the three selected comparison methods on the CarlaEvent3D dataset. Experimental results show that the method of the present invention has a better overall effect compared with the other three methods, thus proving the feasibility of the method proposed in this invention.

Claims

1. A method for three-dimensional motion estimation based on an event camera, characterized in that, comprising the steps of: Step 1, obtaining event sequence and its corresponding pixel-level optical flow label sequence Y O and motion depth label Y D ; thereby constructing event space representation V and event temporal representation K x event temporal representation K yy ; Step 2, construct a spatial encoder based on convolutional neural network for extracting geometric structure features F of event space representation V hw and contextual features F c ; Step 3, construct a one-dimensional convolution-based time sequence encoder, and extract the horizontal time sequence feature F x and the vertical time sequence feature F y from K wt and K ht respectively; Step 4, based on K x and K y and F wt and F ht , construct spatial correlation graph and timing correlation graph ; Step 5, extract the time index sequence t of the top n density maximums along the time axis from the spatiotemporal density vector D l And count the gradient at the corresponding position to get the initial control point P, so as to get the pixel trajectory curve ; Step 6, under the guidance of the pixel trajectory curve , the spatial correlation graph and the timing correlation graph are sampled to construct the spatiotemporal correlation graph C st ; Step 7, the spatiotemporal features F are obtained using formula (11) st ; F st = Concat(C st , F c ) (11) In equation (11), Concat( ) represents the concatenation function; Step 8, calculate the updated control point vector P using formula (12) next ; P next = P + GRU(F st , P) (12) In Formula (12), GRU ) represents a Gated Recurrent Unit; + represents a residual operation; Step 9, update the control point vector P next As P, get the updated control point vector again according to the process of Step 5-Step 8 until the maximum number of updates is reached, so as to get the final control point vector P final Determine the shape of the pixel trajectory curve ​ Step 10, curve fitting for pixel tracks Sampling to get scene optical flow prediction O pred and scene motion depth prediction D pred ; Step 11, based on O pred and Y O and D pred and , construct the loss L, and train the three-dimensional motion estimation network composed of the spatial encoder, the temporal encoder and the gated recurrent unit using the AdamW optimizer, stop training when the loss L is the smallest, thereby obtaining the three-dimensional motion estimation model with optimal parameters, which is used for three-dimensional motion estimation on the input event sequence to realize the prediction of optical flow and motion depth.

2. The event camera based three-dimensional motion estimation method of claim 1, wherein, The step 1 is to construct event space representation V and event timing representation K respectively by using formula (1) and formula (2) x , event timing representation K y ; (1) (2) In formula (1) and formula (2), i represents the serial number of the event; p i , x i , y i , t i respectively represent the polarity, the horizontal axis coordinate, the vertical axis coordinate and the time stamp of the i-th event in V; k( ) is a triangular sampling kernel function; (x, y) is the spatial coordinate of V, and b is the time sequence coordinate of V; is a parameter for controlling the smoothness of the control Gaussian kernel function.

3. The event camera based three-dimensional motion estimation method of claim 2, wherein, The step 4 is to construct a spatial correlation graph using formula (3) - formula (7) and a timing correlation graph : (3) (4) (5) (6) (7) in formulas (3)-(7), represents a Hadamard product; E represents an event sequence ; and D represents a spatiotemporal density vector of the event sequence represents a three-dimensional average pooling operation; D represents a spatiotemporal density vector of the event sequence represents a two-dimensional average pooling operation; represents a one-dimensional average pooling operation; C wt , C ht represent a temporal correlation map along the temporal and horizontal axis dimensions, respectively, and the temporal and vertical axis dimensions, respectively; represents a vector outer product.​​​​ 4. The event camera based three-dimensional motion estimation method of claim 3, wherein, wherein the step 11 is using loss L of formula (13) - formula (16): (13) (14) (15) L = + + (16) In equations (13)-(16), | | is the absolute value operation; This refers to the loss in the optical flow component; This represents the loss in the depth of motion. For time-constrained loss; ( () represents the pixel trajectory curve The derivative of T; T is the sequence of events. Length, represent The timestamp of the (i+1)th event.

5. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store a program supporting the processor to execute the three-dimensional motion estimation method of any one of claims 1-4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to execute the steps of the three-dimensional motion estimation method of any one of claims 1-4.

Images