A high-speed high-maneuver three-dimensional trajectory prediction method and system combining event imaging and motion decoupling

Abstract

The present application belongs to the technical field of trajectory prediction, and discloses a high-speed high-maneuver three-dimensional trajectory prediction method and system combining event imaging and motion decoupling. The method comprises the following steps: S1, converting an event stream collected at any target time into an event stream at a reference time; S2, calculating the three-dimensional space coordinates corresponding to each target object in the event stream at the reference time; S3, repeating S2 to obtain the three-dimensional space coordinates of each target object in different time windows, i.e. the three-dimensional motion trajectory of each target object; S4, calculating the motion distance and motion direction of the prediction point by using the first plurality of trajectory points in the three-dimensional motion trajectory, thereby realizing trajectory prediction. Through the present application, the problems of difficult three-dimensional trajectory prediction and low prediction accuracy caused by the fast motion speed and random direction of high-speed high-maneuver targets are solved.

Description

Technical Field

[0001] This invention belongs to the field of trajectory prediction technology, and more specifically, relates to a high-speed, high-maneuverability three-dimensional trajectory prediction method and system that combines event imaging and motion decoupling. Background Technology

[0002] An event camera is a neuromorphic vision sensor. Compared to a frame camera that acquires images at a fixed frame rate, an event camera asynchronously triggers events based on changes in pixel light intensity. It offers advantages such as fast response, wide dynamic range, low bit rate, and low power consumption, making it a promising candidate for applications in fields such as drone obstacle avoidance and space target surveillance. For example, vision-based drone obstacle avoidance requires detecting flying obstacles and responding quickly; vision-based space target surveillance requires perceiving space targets in complex scenarios such as high speed and high maneuverability.

[0003] Patents CN202110101658.1, CN202310328655.0, and CN202310381363.3 utilize point clouds and image processing to detect moving targets, correlate the detection results with the predicted trajectory, and update the predicted trajectory. However, these methods fail in high-speed, high-maneuverability scenarios. The Sato team at Chiba University (IROS2020) uses high-order polynomials to predict target motion trends by fitting position distributions; however, the essence of high-order polynomial fitting is to model the direction changes of the moving target in a constant manner. The Forrai team at ETH Zurich (ICRA2023) uses parabolic equations based on initial velocity and gravitational acceleration to predict the future motion trend of the target; however, the essence of parabolic equation prediction is to model the velocity changes of the moving target in a constant manner. These methods of modeling target motion changes in a constant manner are ineffective for highly maneuverable targets because the maneuverability of the moving target breaks this constant motion change pattern, exhibiting complex motion variations. The challenge of the aforementioned application tasks lies in the irregular changes in the speed and direction of moving targets in high-speed, high-maneuverability scenarios. This makes it impossible for existing methods to accurately predict the three-dimensional trajectory of high-speed, high-maneuverability targets. Summary of the Invention

[0004] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a high-speed, high-maneuverability three-dimensional trajectory prediction method and system that combines event imaging and motion decoupling, thereby solving the problem of low accuracy in high-speed, high-maneuverability trajectory prediction.

[0005] To achieve the above objectives, according to one aspect of the present invention, a high-speed, high-maneuverability three-dimensional trajectory prediction method combining event imaging and motion decoupling is provided, the method comprising the following steps:

[0006] S1 converts the pixel coordinates of each event point in the event stream collected at any target time period into the pixel coordinates of the event point at the reference time period, thereby obtaining the event stream of the reference time period;

[0007] S2 calculates the average timestamp of each event point in the event stream of the reference time period within a preset time window z and normalizes the average timestamp; the average timestamps of all event points after normalization are used to form a timestamp image; targets in the timestamp image are identified and all targets are clustered into multiple categories, with each category being a target object; the two-dimensional coordinates of each target object in the timestamp image are projected to three-dimensional space to obtain the three-dimensional spatial coordinates of each target object;

[0008] S3 z=z+1, repeat S2 until the maximum number of time windows, obtain the three-dimensional spatial coordinates of each target object in all time windows within the reference time period, that is, obtain the three-dimensional motion trajectory of each target object.

[0009] S4 uses the previous multiple trajectory points in the three-dimensional motion trajectory to calculate the motion distance and direction to the next trajectory point, thereby predicting the next trajectory point and achieving trajectory prediction.

[0010] More preferably, in step S4, the movement distance is calculated according to the following formula:

[0011]

[0012] in, is the predicted motion vector magnitude, and n is the number of historical trajectory points used to predict the motion vector magnitude. It is the motion vector magnitude at the i-th historical moment; is the weight of the motion vector magnitude of the i-th trajectory point, and m is the number of trajectory points used to balance noise. c , d All are predefined linear parameters.

[0013] More preferably, in step S4, the direction of motion is calculated according to the following formula:

[0014]

[0015] in, It is the predicted direction vector of motion. It is the direction vector of motion at the current moment; It is the angle of change of direction; It is geometric curvature. It is the predicted motion vector model The value of .

[0016] More preferably, in step S4, the formula for calculating the position of the next trajectory point is as follows:

[0017]

[0018] in, It is a predicted location. This is the current location. It is the predicted direction vector of motion. It is the predicted motion vector magnitude.

[0019] More preferably, in step S2, the three-dimensional spatial coordinates of the target object are obtained according to the following steps:

[0020] Calculate the target depth for each target object;

[0021] Project the target object onto the camera coordinate system in three-dimensional space;

[0022] The target object is transformed from the camera coordinate system to the world coordinate system to obtain the three-dimensional spatial coordinates of the target object.

[0023] More preferably, the formula for calculating the target depth is as follows:

[0024]

[0025] Where f is the focal length of the event camera. It is the system baseline for event cameras. and It refers to the x-axis coordinates of the same target in the images obtained by the left and right event cameras. It refers to the target depth.

[0026] More preferably, the camera coordinate system projected onto the target object in three-dimensional space is calculated according to the following formula:

[0027]

[0028] The target object is transformed from the camera coordinate system to the world coordinate system using the following formula:

[0029]

[0030] in, These are the three-dimensional coordinates of an object in the world coordinate system. It is the transformation matrix from the camera coordinate system to the world coordinate system. These are three-dimensional coordinates in the camera coordinate system.

[0031] More preferably, in step S2, the formulas for calculating the planar timestamp and the normalization process are as follows:

[0032]

[0033]

[0034] in, It is a pixel Average timestamp image; It is a subset of events The number of events in; It is an event timestamp, It is a normalized average timestamp image; It is the mean of the average timestamp image T; It is the time interval for calculating the timestamp image.

[0035] More preferably, in step S2, the determination of the target in the timestamp image is performed according to the following steps:

[0036] (1) Calculate the target segmentation threshold according to the following formula;

[0037]

[0038] in, It is the threshold for separating the target from the background. It is a normalized timestamp image N The mean; It is the angular velocity of the event camera; It is the linear velocity of the event camera; a , b , p All of these are predefined parameters.

[0039] (2) Determine the difference between the normalized average timestamp of each event point and the segmentation threshold. Event points that are greater than the segmentation threshold are the target, and others are the background.

[0040] According to another aspect of the present invention, a high-speed, high-maneuverability three-dimensional trajectory prediction system combining event imaging and motion decoupling is provided. The system includes an actuator for performing the aforementioned high-speed, high-maneuverability three-dimensional trajectory prediction method combining event imaging and motion decoupling.

[0041] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0042] 1. This invention provides a high-speed, high-maneuverability 3D trajectory prediction method that combines event imaging and motion decoupling. By processing the event stream at different target times, the motion trajectory of the target object is obtained, ensuring the motion continuity of the moving target. It supports accurate modeling of motion changes based on continuous motion, thereby improving the performance of 3D trajectory prediction and alleviating the problems of difficulty in 3D trajectory prediction and low prediction accuracy caused by the high speed of high-speed target motion.

[0043] 2. This invention provides a high-speed, high-maneuverability three-dimensional trajectory prediction method that combines event imaging and motion decoupling. By decoupling the motion of a moving target into two dimensions, motion distance and motion direction, and by predicting and fusing these two dimensions separately, the accuracy of the three-dimensional trajectory prediction of the moving target is ensured, thus improving the problem of low accuracy in three-dimensional trajectory prediction caused by the complex and variable motion trajectory of highly maneuverable targets. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating a high-speed, high-maneuverability three-dimensional trajectory prediction method that combines event imaging and motion decoupling, constructed according to a preferred embodiment of the present invention.

[0045] Figure 2 Figure 1 shows a comparison of the trajectory acquisition capabilities of an event camera and a traditional frame camera at different motion speeds, constructed according to a preferred embodiment of the present invention. Figure 2 shows a visual comparison between the dense event stream acquired by the event camera and the sparse image frames acquired by the traditional frame camera; Figure 3 shows the target trajectory acquired by the frame camera in a low-speed scene; Figure 4 shows the target trajectory acquired by the event camera in a low-speed scene; Figure 5 shows the target trajectory acquired by the frame camera in a high-speed scene; Figure 6 shows the target trajectory acquired by the event camera in a high-speed scene.

[0046] Figure 3 Figure (a) shows the characteristics of maneuvering and non-maneuvering targets, constructed according to a preferred embodiment of the present invention. Figure (b) shows the velocity prediction paradigm, Figure (c) shows the direction prediction paradigm, and Figure (d) shows the velocity and direction prediction paradigm.

[0047] Figure 4 This is a detailed implementation diagram of the three-dimensional trajectory prediction of a high-speed, highly maneuverable unmanned aerial vehicle target constructed according to a preferred embodiment of the present invention;

[0048] Figure 5 This refers to the acquisition device and data sample constructed according to a preferred embodiment of the present invention. Figure (a) shows the acquisition device, including a binocular stereo event camera, RTK, and a drone. Figure (b) shows the data sample, containing a stereo event stream and drone trajectory; the actual trajectory is acquired via RTK.

[0049] Figure 6 This section discusses the advantages of event cameras over traditional frame cameras for UAV targets at different speeds. Figure (a) shows a comparison of the trajectory continuity between traditional frame cameras and event cameras, while Figure (b) shows a comparison of the minimum average displacement error between traditional frame cameras and event cameras.

[0050] Figure 7 It is a visual comparison of trajectory prediction for moving targets using different maneuvers. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0052] like Figure 1 As shown, a high-speed, high-maneuverability 3D trajectory prediction method combining event imaging and motion decoupling is proposed. The method includes the following steps:

[0053] S1, Obtain the event stream at the reference time.

[0054] Motion compensation: Using inertial measurement information, a motion compensation strategy is used to align event flows from different times to their spatial positions at the same time.

[0055] The inertial measurement unit and event camera mounted on the moving platform are jointly calibrated to determine the transformation relationship between the event camera coordinate system and the inertial measurement unit coordinate system, and to obtain the external parameter matrix.

[0056] The rotation matrix is ​​calculated using an inertial measurement unit, and the pixel coordinates of each event point in the event stream acquired by the event camera at the target time are converted into the pixel coordinates of the event point at the reference time. This achieves motion compensation of the event stream to obtain the event stream at the reference time.

[0057] Motion compensation essentially involves transforming the pose between adjacent time stamps, using geometric projection relationships to transform the event pixel coordinates at the target time to the reference time. Given an event stream over a certain time period, and measuring the inertial angular velocity of a specific event and the event camera, the rotation matrix R of the event relative to the initial reference time is calculated using the Rodrigues formula:

[0058]

[0059] in, , , It is the average angular velocity over the same time period. . , and It is the rotation matrix calculated using Rodrigues' rotation formula.

[0060] Rotation matrix from IMU to event camera Rotational motion compensation is achieved through the following geometric projection:

[0061]

[0062] in, Fix the camera parameters. The event coordinates before motion compensation. The coordinates of the event after motion compensation.

[0063] The key to motion compensation lies in using inertial measurement information to align the pixel spatial positions of background events in the event stream, thereby facilitating the extraction of moving targets within the events. When performing motion compensation on the raw event stream captured by the event camera, the selectable time interval covers the range of 3ms to 50ms, making it suitable for tasks with high real-time requirements.

[0064] S2. Calculate the three-dimensional spatial coordinates of each target object in the event stream at the reference time.

[0065] Specifically, the average timestamp of each event point within a preset time window z of the event stream during the reference time period is calculated and normalized. The average timestamps of all event points after normalization form a timestamp image. Targets in the timestamp image are identified and clustered into multiple categories, each category representing a target object. The two-dimensional coordinates of each target object in the timestamp image are projected into three-dimensional space to obtain the three-dimensional spatial coordinates of each target object. Target detection based on threshold segmentation: The motion-compensated event stream is used to construct a timestamp image, which is then normalized. Finally, an adaptive thresholding strategy is used to segment targets from the normalized timestamp mean image, thereby extracting independent moving targets. The core process in this step is constructing timestamp images to eliminate camera motion interference, using event time characteristics for adaptive threshold segmentation, followed by DBSCAN clustering to extract targets, and using EKF tracking to ensure continuity. The specific steps are as follows:

[0066] S21. Construct a timestamp image:

[0067] (1) Set a time window on the event stream at the reference time to obtain the set of event points in that time window;

[0068] (2) Calculate the average timestamp of each event point in the event point set to obtain an average timestamp image formed by the average timestamps of all event points;

[0069] (3) Normalize the average timestamp image to obtain the timestamp image. First, normalize the event set after motion compensation. Events are grouped according to their positions on the image plane. For each pixel position, the average timestamp of all events at that position is calculated and then normalized.

[0070] Formula for calculating the event count pixels for each event and generating a timestamp image:

[0071]

[0072] in, Indicates that it is located in the event set The subset of pixels is located at (g, h). Then, an average timestamp image is constructed by calculating the average timestamp for each event point, using the following formula:

[0073]

[0074] in It is a pixel Average timestamp image; It is a subset of events The number of events in; It is an event timestamp;

[0075] By normalizing the average timestamp image, the normalized average timestamp is calculated over a fixed time interval as follows:

[0076]

[0077] in, It is a normalized average timestamp image; It is the mean of the average timestamp image T; It is the time interval for calculating the timestamp image.

[0078] S22. Target Segmentation: An adaptive thresholding strategy is used to segment the target from the normalized timestamp mean image N. Assume the angular velocity of the event camera is... The linear velocity is The segmentation threshold is:

[0079]

[0080] Where a, b, and p are predefined parameters. (Using a threshold) The target and background are classified: those larger than the segmentation threshold are considered targets, and those not are considered background.

[0081] Target mapping and tracking based on DBSCAN clustering algorithm and extended Kalman filter: The DBSCAN clustering algorithm is used to process the normalized average timestamp image segmentation results to obtain target clusters, bounding boxes, and cluster centers to distinguish target objects. Then, the extended Kalman filter is used to predict the historical target positions, and the Mahalanobis distance is combined to match the cluster centers of the current frame to achieve target mapping. At the same time, its prediction-update mechanism is used to ensure the continuity of target motion state and obtain continuous two-dimensional target trajectories. In this step, the data in the first N time windows can be obtained by splitting the data within a single time window, where N is a predefined parameter.

[0082] S23. Target Clustering: For cases where the normalized average timestamp image may contain multiple objects, the DBSCAN clustering algorithm is used to cluster the segmentation results of the normalized average timestamp image to distinguish instances of each object. Subsequently, the largest bounding boxes around these clusters are fitted.

[0083] Specifically, the segmented target pixel coordinates are processed by the DBSCAN clustering algorithm to obtain multiple target clusters, the bounding box of each cluster, and the coordinates of the cluster center.

[0084] S24. Target Correspondence and Tracking: For multiple target objects after DBSCAN clustering, the two-dimensional position of each target at different times must be associated with the "motion trajectory of the same target," otherwise target trajectories will be broken or confused. This step is solved using the Extended Kalman Filter (EKF) and Mahalanobis distance data association framework.

[0085] Specifically,

[0086] (1) Target correspondence: First, the event stream acquired by the event camera after motion compensation is processed by the DBSCAN clustering algorithm to obtain the coordinates of the two-dimensional cluster centers of each target in the current frame; second, the state prediction of the tracked historical targets is performed based on EKF, and the predicted position and prediction covariance matrix of each historical target are output; finally, the Mahalanobis distance between the DBSCAN cluster center and the predicted position in the current frame is calculated, and effective matching is achieved by the minimum distance criterion, that is, the minimum Mahalanobis distance is selected. d min ,like d min If the values ​​are less than a preset threshold, the two are determined to be related and are the same object. This establishes a cross-time correspondence for the same target, thereby achieving accurate matching of each target.

[0087] (2) Target tracking: used to maintain the continuity of the target's motion state, specifically including: First, for the effectively associated target, initialize the EKF state, the initial value is calculated from the cluster center coordinates and the position difference between adjacent frames; Second, dynamically update the target state based on the EKF prediction-update mechanism: predict the current position according to the assumption that "the target maintains uniform motion in a short time", combine the newly detected position to correct the prediction value, and output a smooth two-dimensional image coordinate sequence of the same target at different times, so as to provide continuous and accurate motion state input for subsequent trajectory prediction and ensure that the trajectory is unbroken.

[0088] 3D geometric projection reconstruction based on binocular vision: First, the disparity is calculated through stereo matching. The target depth is obtained by using the focal length, baseline, and coordinate difference between the left and right images according to the common depth calculation formula. Then, the target coordinates are projected onto the camera coordinate system by combining the camera intrinsic parameters. Finally, the target's 3D motion trajectory is reconstructed by transforming it to the real-world coordinate system using a transformation matrix.

[0089] S25. Depth estimation: The depth of the target is calculated using a stereo matching strategy, and the depth is estimated by using the disparity between corresponding points in paired stereo images.

[0090] The formula for calculating the target depth is as follows:

[0091]

[0092] Where f is the focal length of the event camera. It is the baseline of the stereo camera system. and These are the x-axis coordinates of the same target in the 3D image. It refers to the target depth.

[0093] S26. Geometric Projection Reconstruction: This method reconstructs the 3D trajectory of a target by projecting its center on the image plane into 3D space. Given the inherent parameters K of the event camera and the depth of the target... By applying projection from two dimensions to three dimensions:

[0094]

[0095] These are the coordinates of the target in the timestamp image. These are coordinates in the camera coordinate system.

[0096] S27, will Project onto a real-world coordinate system to establish a connection with the real-world coordinate system:

[0097]

[0098] in It is the transformation matrix from image coordinates to real-world coordinates. This is the target's final position in real-world coordinates. The target's velocity in the real world can be calculated by reconstructing the distance and time interval between two points in the trajectory.

[0099] S3, z=z+1, repeat S2 until the maximum number of time windows, obtain the three-dimensional spatial coordinates of each target object in all time windows within the reference period, that is, obtain the coordinates of each target object at all times, thereby forming the three-dimensional motion trajectory of the target object and realizing the reconstruction of the three-dimensional trajectory of the moving target.

[0100] Specifically,

[0101] Let the time window index z increment from its initial value, and repeat step S2 for all time windows within the reference period: For each time window, construct a normalized average timestamp image from the motion-compensated event stream, extract target pixels using adaptive threshold segmentation, obtain the two-dimensional cluster centers of each target through DBSCAN clustering, and combine extended Kalman filtering to achieve target correspondence and tracking across time windows to ensure motion continuity. Then, based on depth estimation and geometric projection transformation of binocular vision, calculate the three-dimensional spatial coordinates of each target in the real-world coordinate system within the time window. When z traverses to the maximum number of time windows, each target object will obtain a sequence of three-dimensional coordinates corresponding to all time windows within the reference period in chronological order. This sequence constitutes the complete motion trajectory of the target object, thus completing the reconstruction of the three-dimensional trajectory of the moving target.

[0102] S4. The motion distance and direction of the predicted point are calculated using the first few trajectory points in the 3D motion trajectory, thus achieving trajectory prediction. A motion decoupling strategy and a motion fusion strategy are proposed to predict the target's motion trajectory. Based on the different characteristics of the moving target in different dimensions, the target's trajectory prediction is first decoupled into component predictions of different dimensions, and then the prediction results of each component are fused to obtain the 3D trajectory prediction result of the moving target.

[0103] This includes motion decoupling and motion fusion. Details are as follows:

[0104] First, the trajectory prediction of high-speed, highly maneuverable targets is decoupled into distance prediction and direction prediction. A linear weighted model and a weight-aware strategy are used to model the change in the target's distance, while geometric curvature is used to model the change in the target's direction. Then, vector operations are used to fuse the predicted distance and direction. This solves the problem of difficulty in 3D trajectory prediction caused by the high speed and complex, variable trajectory of the target.

[0105] S41. Motion Decoupling: Because moving targets exhibit continuous motion, the angle of change in their motion direction follows local consistency over a short period, and the motion distance conforms to the principle of differentiability of a continuous linear model. Therefore, the trajectory prediction of high-speed, highly maneuverable targets is decoupled into motion distance prediction and motion direction prediction, specifically including:

[0106] S411, Motion Distance Prediction: The predicted motion distance is the sum of the distance change and the base distance. A linear model is introduced to represent the change in motion distance of the target at the high temporal resolution of the event camera, and a weight-aware accumulation strategy is used to model the base distance.

[0107] This method, based on the uncertainty noise generated in the target detection and 3D trajectory reconstruction modules, linearly models the average distance change over a short period of time, as shown in the following formula:

[0108]

[0109] in, is the motion vector magnitude at the i-th historical moment; m is the number of trajectory points used to balance noise. c , d All are predefined linear parameters. The distance between the predicted location and the original location is the sum of the distance change and the base distance.

[0110] A weight-aware accumulation strategy is used to model the base distance, as shown in the following formula:

[0111]

[0112] in, These are the weights of the predicted motion vector magnitude of the i-th trajectory point. It is the motion vector magnitude at the i-th historical moment; is the predicted basic motion vector magnitude, i.e., the predicted basic distance; n is the number of historical trajectory points used to predict the basic distance.

[0113] Combining the above two equations, we obtain the final predicted motion vector, which is the final predicted distance modulus. :

[0114]

[0115] Furthermore, in the above formula, n, m, The value can be chosen by the user or learned using a deep learning network.

[0116] S412, Direction Prediction: Since the angle of change in the direction of a moving target follows local consistency over a short period, this method introduces a high temporal resolution event camera to ensure motion continuity and satisfy local consistency. Then, geometric curvature is used to represent the direction change, thereby achieving direction prediction. Specifically, by specifying the three most recent points in the past trajectory, namely p1, p2, and p3, as reference points, such as... Figure 4 The gray dot is shown in the image. First, the geometric curvature of the reference point is calculated as follows:

[0117]

[0118] Where k is the geometric curvature of the reference point. Indicates the origin of the point With point The constructed three-dimensional vector, symbol This represents the cross product operation. This represents vector norm operations. Then, the previously predicted motion vector norm is used... Considered as the arc length of the predicted trajectory, defined by curvature Calculate the change angle of the moving target's direction. Using the Rodrigues rotation formula, the direction prediction of highly maneuverable targets is achieved by rotating the tangent at the current position.

[0119]

[0120]

[0121] in, It represents the direction between the current position and the predicted position.

[0122] S42. Motion Fusion: Utilizing the predicted distance and direction from the proposed motion decoupling strategy, fusion is achieved through vector operation criteria, thereby realizing the three-dimensional trajectory prediction of high-speed, highly maneuverable targets. The fusion formula is as follows:

[0123]

[0124] in, It is a predicted location. This is the current location. It is the predicted direction vector of motion. It is the predicted motion vector magnitude, i.e., the predicted final distance.

[0125] The following will provide further explanation with reference to specific embodiments.

[0126] First, a binocular event camera system was established for data collection. Then, a drone was used as a high-speed, highly maneuverable object to simulate motion trajectories with various changes by performing complex maneuvering scenarios. This was used to experimentally verify the method of the present invention. The experiment was conducted within a range of 3-10 meters from the binocular event camera system, with the drone's maximum flight speed being approximately 14 meters per second. The drone's actual trajectory was obtained using an RTK system installed on the drone, as detailed below:

[0127] (1) Data classification: The collected trajectories are divided into three categories: stable segment (the UAV moves at an almost constant speed without changing direction), variable speed segment (the UAV’s speed changes irregularly, but the direction remains unchanged), and variable speed maneuver segment (there are obvious turning movements and speed changes).

[0128] (2) Experimental results: The results show that by introducing an event camera, the motion continuity of the moving target can be guaranteed, and the motion change can be accurately modeled on the basis of continuous motion, thereby improving the performance of the three-dimensional trajectory prediction of the target and alleviating the problem of the difficulty in predicting the three-dimensional trajectory caused by the high speed of high-speed target motion. Under the non-motion decoupling strategy, the trajectory prediction is inaccurate. By decoupling the motion of the moving target into two dimensions, motion distance and motion direction, and by predicting and fusing these two dimensions separately, the accuracy of the three-dimensional trajectory prediction of the moving target is guaranteed, and the problem of the difficulty in predicting the three-dimensional trajectory caused by the complex and variable motion trajectory of highly maneuverable targets is improved.

[0129] like Figure 2 As shown, this is a comparison of the trajectory acquisition capabilities of an event camera and a traditional frame camera for moving targets at different speeds, constructed according to a preferred embodiment of the present invention. Figure (a) shows an observation scene on the left of a high-speed, highly maneuverable target crossing the camera's field of view, and on the right, a visual comparison of the dense event stream acquired by the event camera and the sparse image frames acquired by the traditional frame camera. The traditional frame camera samples in discrete frames, and the red target trajectory is segmented into discrete fragments due to frame intervals. The event camera outputs an asynchronous event stream, and the target trajectory is continuously and densely distributed. Figures (b, c, d, e) are divided according to target speed and camera type. Figure (b) shows the target trajectory acquired by the frame camera in a low-speed scene; Figure (c) shows the target trajectory acquired by the event camera in a low-speed scene; Figure (d) shows the target trajectory acquired by the frame camera in a high-speed scene; and Figure (e) shows the target trajectory acquired by the event camera in a high-speed scene.

[0130] In scenarios with low target speeds, while traditional frame cameras can capture dense trajectories, event cameras offer superior continuity. In scenarios with high target speeds, traditional frame cameras produce sparse, discrete trajectories due to large target displacements within the sampling interval, whereas event cameras still output dense, continuous trajectories. This comparison demonstrates the high temporal resolution advantage of event cameras, making them more robust for target trajectory reconstruction and fully showcasing their irreplaceable role in high-speed, high-maneuverability scenarios.

[0131] like Figure 3 As shown in the figure, this paper focuses on the verification of trajectory prediction methods for non-motorized and motorized objects. Figure 3 (a) Targets are classified according to their motion characteristics: First, non-motorized objects with approximately constant initial velocity and direction, dominated by gravity; second, motorized objects with dynamically changing velocity and direction. Figure 3 (b) is a parabolic model. The model assumes "constant velocity and constant gravitational acceleration". In the non-motorized object scenario, the predicted point fits the actual trajectory because it conforms to the projectile motion assumption. However, for the motorized object, the prediction deviates completely due to the dynamic changes in velocity and direction. Figure 3 In equation (c), a high-order polynomial fitting model is used, which assumes a "constant direction of motion." In non-motorized object scenarios, the gradual change in direction allows for a certain level of accuracy, while in motorized object scenarios, dynamic changes in direction lead to prediction failure. The method proposed in this invention... Figure 3 The motion decoupling method in (d) decomposes motion into velocity and direction, which are modeled independently. This strategy is suitable for the slow-changing characteristics of non-motorized objects and can respond to the sudden changes in velocity and direction of motorized objects. This verifies the universality of motion decoupling for scenarios with varying speed and direction.

[0132] like Figure 4 As shown in the figure, trajectory prediction mainly includes three parts: target detection, trajectory reconstruction, and trajectory prediction. Given an input stereo event stream, we locate the target on the image plane through target detection, and then use stereo vision technology to project the moving target into a three-dimensional coordinate system for trajectory reconstruction. Furthermore, we predict the motion trend of the moving target in the velocity and direction dimensions through motion decoupling, and then fuse the prediction results of velocity and direction through vector operations to achieve trajectory prediction.

[0133] like Figure 5 (a) shows the data acquisition equipment, which includes a drone, an RTK positioning system, and a stereo event camera. The drone, as a high-speed and highly maneuverable target, performs complex maneuvers such as acceleration and turning according to a preset trajectory. The RTK positioning module is integrated into the drone and outputs high-precision global position and attitude data as a real reference benchmark for trajectory prediction. The stereo event camera captures the asynchronous event stream of the drone's motion. Figure 5(b) shows the collected data: first, the left and right event frames output by the binocular event camera, which fully preserve the temporal continuity of the target's motion; second, the 3D trajectory of the UAV generated by RTK, which, through normalized velocity, intuitively presents the velocity changes and maneuver characteristics of the trajectory.

[0134] like Figure 6 (a) qualitatively demonstrates that for moving targets, especially high-speed moving targets, traditional frame cameras can only capture sparse trajectories, while event cameras can capture denser, more continuous trajectories. Figure 6 (b) in the diagram can be quantitatively verified that for high-speed objects, event cameras offer higher positioning accuracy and smaller average minimum displacement error than traditional frame cameras, with a more significant advantage in high-speed scenes. Combined, these findings indicate that event cameras are better suited for the localization and prediction of dynamic, high-speed moving targets.

[0135] like Figure 7 As shown, the results of three-dimensional trajectory prediction under different maneuvering characteristics are presented. For non-maneuvering targets with smooth trajectories and gradual speed changes, and for maneuvering targets with different speeds such as low speed and high speed, the prediction effects of support vector regression, higher-order polynomial, parabolic equation methods and the method of this invention are compared.

[0136] In non-motorized scenarios, other methods struggle to accurately fit long-term stable motion patterns, resulting in deviations between predicted and actual trajectories. In low-motorized scenarios, other methods are ill-suited to small changes in speed and direction, leading to divergent predicted trajectories. In high-motorized scenarios, other methods fail entirely due to their reliance on the "constant motion assumption," while the method of this invention utilizes the high temporal resolution of an event camera to ensure motion continuity and employs "motion decoupling modeling" to achieve a close fit between predicted and actual trajectories. The results in this figure visually demonstrate the superior prediction accuracy of the method across the entire range of motorized scenarios, particularly its performance breakthrough in high-motorized scenarios.

[0137] This invention employs a method to predict the 3D trajectory of a moving target based on the continuity of motion during imaging. This method models the changes in the target's distance and direction of motion using a combined event imaging system and a motion decoupling strategy, thereby achieving prediction of the future motion trend of high-speed, highly maneuverable targets. Specifically, an event camera is introduced to ensure the continuity of the target's motion, guaranteeing accurate modeling of motion changes based on continuous motion. Furthermore, the trajectory prediction of high-speed, highly maneuverable targets is further decoupled into distance prediction and direction prediction. A linear weighted model is used to model the target's distance changes, and geometric curvature is used to model the target's direction changes. Finally, vector operation rules are used to fuse the predicted distance and direction. This solves the problem of difficulty in 3D trajectory prediction caused by the high speed and complex, variable trajectory of the target. The method in this invention ensures the continuity of high-speed moving targets by introducing the high temporal resolution of an event camera, decouples the trajectory prediction of highly maneuverable targets into distance prediction and direction prediction using the proposed motion decoupling strategy, and finally fuses the predicted distance and direction using vector operation rules, thereby achieving 3D trajectory prediction of high-speed, highly maneuverable targets.

[0138] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A high-speed, high-maneuverability three-dimensional trajectory prediction method combining event imaging and motion decoupling, characterized in that, The method includes the following steps: S1 converts the pixel coordinates of each event point in the event stream collected at any target time period into the pixel coordinates of the event point at the reference time period, thereby obtaining the event stream of the reference time period; S2 calculates the average timestamp of each event point in the event stream of the reference period within the preset time window z and normalizes the average timestamp. The average timestamps after normalization of all event points form a timestamp image; Identify targets in the timestamp images and cluster all targets into multiple categories. Use an adaptive threshold strategy to segment targets from the normalized timestamp mean image N. Targets larger than the segmentation threshold are considered targets, otherwise they are considered background. Each category is treated as a target object. For multiple target objects after clustering, the two-dimensional position of each target at different times must be associated with the motion trajectory of the same target. The depth of the target is calculated by a stereo matching strategy, and the two-dimensional coordinates of each target object in the timestamp image are projected to three-dimensional space to obtain the three-dimensional spatial coordinates of each target object in the real world coordinate system. S3 z=z+1, repeat S2 until the maximum number of time windows, obtain the three-dimensional spatial coordinates of each target object in all time windows within the reference time period, that is, obtain the three-dimensional motion trajectory of each target object. S4 uses the previous multiple trajectory points in the three-dimensional motion trajectory to calculate the motion distance and direction of the next trajectory point, thereby predicting the next trajectory point and thus achieving trajectory prediction. In step S4, the movement distance is calculated according to the following formula: in, is the predicted motion vector magnitude, and n is the number of historical trajectory points used to predict the motion vector magnitude. It is the motion vector magnitude at the i-th historical moment; is the weight of the motion vector magnitude of the i-th trajectory point, and m is the number of trajectory points used to balance noise. c , d All are predefined linear parameters; In step S4, the direction of motion is calculated according to the following formula: in, It is the predicted direction vector of motion. It is the direction vector of motion at the current moment; It is the angle of change of direction; It is geometric curvature. It is the predicted motion vector model The value of .

2. The high-speed, high-maneuverability three-dimensional trajectory prediction method combining event imaging and motion decoupling as described in claim 1, characterized in that, In step S4, the formula for calculating the position of the next trajectory point is as follows: in, It is a predicted location. This is the current location. It is the predicted direction vector of motion. It is the predicted motion vector magnitude.

3. The high-speed, high-maneuverability three-dimensional trajectory prediction method combining event imaging and motion decoupling as described in claim 1, characterized in that, In step S2, the three-dimensional spatial coordinates of the target object are obtained according to the following steps: Calculate the target depth for each target object; Project the target object onto the camera coordinate system in three-dimensional space; The target object is transformed from the camera coordinate system to the world coordinate system to obtain the three-dimensional spatial coordinates of the target object.

4. The high-speed, high-maneuverability three-dimensional trajectory prediction method combining event imaging and motion decoupling as described in claim 3, characterized in that, The formula for calculating the target depth is as follows: Where f is the focal length of the event camera. It is the system baseline for event cameras. and It refers to the x-axis coordinates of the same target in the images obtained by the left and right event cameras. It refers to the target depth.

5. A high-speed, high-maneuverability three-dimensional trajectory prediction method combining event imaging and motion decoupling as described in claim 3 or 4, characterized in that, The target object projected onto the camera coordinate system in three-dimensional space is calculated according to the following formula: The target object is transformed from the camera coordinate system to the world coordinate system using the following formula: in, These are the three-dimensional coordinates of an object in the world coordinate system. It is the transformation matrix from the camera coordinate system to the world coordinate system. These are three-dimensional coordinates in the camera coordinate system.

6. The high-speed, high-maneuverability three-dimensional trajectory prediction method combining event imaging and motion decoupling as described in claim 1, characterized in that, In step S2, the formulas for calculating the planar timestamp and the normalization process are as follows: in, It is a pixel Average timestamp image; It is a subset of events The number of events in; It is an event timestamp, It is a normalized average timestamp image; It is the mean of the average timestamp image T; It is the time interval for calculating the timestamp image.

7. The high-speed, high-maneuverability three-dimensional trajectory prediction method combining event imaging and motion decoupling as described in claim 1, characterized in that, In step S2, the determination of the target in the timestamp image is performed according to the following steps: (1) Calculate the target segmentation threshold according to the following formula; in, It is the threshold for separating the target from the background. It is a normalized timestamp image N The mean; It is the angular velocity of the event camera; It is the linear velocity of the event camera; a , b , p All are predefined parameters; (2) Determine the difference between the normalized average timestamp of each event point and the segmentation threshold. Event points that are greater than the segmentation threshold are the target, and others are the background.

8. A high-speed, high-maneuverability three-dimensional trajectory prediction system combining event imaging and motion decoupling, characterized in that, The system includes an actuator for performing a high-speed, high-maneuverability three-dimensional trajectory prediction method combining event imaging and motion decoupling as described in any one of claims 1-7.

Images