A high-speed motion three-dimensional point tracking and trajectory reconstruction method and system

Abstract

The application discloses a high-speed motion three-dimensional point tracking and trajectory reconstruction method and system, and relates to the technical field of three-dimensional point tracking. The method comprises the following steps: synchronously collecting an RGB image sequence and an event stream through a binocular multi-modal observation device; performing time alignment on the RGB image sequence and the event stream, extracting an RGB feature map based on the RGB image sequence, and extracting an event feature map based on the event stream; mapping the RGB feature map and the event feature map to a three-dimensional space; supplementing a three-dimensional feature point cloud through a linear interpolation algorithm; balancing the spatial distribution of the supplemented three-dimensional feature point cloud through an anchor point sampling algorithm to obtain a fused three-dimensional feature point cloud; and generating high-speed three-dimensional point trajectory data based on the fused three-dimensional feature point cloud through a time correlation transformer. The application effectively fuses the texture fidelity of the RGB image and the high time resolution of the event data, and can realize high-frame-rate and high-precision three-dimensional point trajectory reconstruction in a high-speed motion scene.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional point tracking technology, and in particular to a method and system for high-speed motion three-dimensional point tracking and trajectory reconstruction. Background Technology

[0002] 3D point tracking is one of the core problems in computer vision. Its goal is to estimate the temporally coherent motion trajectory of any point in 3D space through continuous visual observation. Compared to 2D point tracking, 3D point tracking can record motion details from a more comprehensive spatial perspective, which is crucial for understanding complex motion patterns and has significant value in applications such as robot manipulation, 3D reconstruction, and dynamic scene analysis.

[0003] Significant progress has been made in existing technologies for 3D point tracking based on stereo vision. In monocular setups, existing techniques can elevate 2D points to 3D space for tracking through a three-plane representation; or represent the input video as a multi-scale 2D feature map and fuse it with 3D information for enhancement; or employ a coarse-to-fine estimation strategy to achieve dense point trajectory estimation on the image plane. In multi-camera applications, techniques can also achieve online 3D point tracking using synchronized RGB frames from multiple viewpoints; or achieve high-precision 2D and 3D point tracking in multi-camera systems through a time-varying 3D Gaussian reconstruction model.

[0004] On the other hand, event cameras, as neuromorphic vision sensors, respond asynchronously to changes in pixel brightness, offering advantages such as microsecond-level temporal resolution, high dynamic range, and low power consumption. Researchers have applied event cameras to stereo vision tasks. Examples include reconstructing 3D representations from event streams, fusing events with blurred frames to achieve high-quality reconstruction, and realizing dynamic reconstruction from multi-view event cameras. These studies have driven the application of 3D point tracking in event stereo vision.

[0005] However, most current 3D point tracking methods are still limited to ordinary low-speed motion scenarios, making it difficult to accurately recover 3D point trajectories in real high-speed motion phenomena. The key bottleneck lies in perception technology: conventional RGB cameras have a low frame rate (approximately 30 frames per second), which easily produces motion blur and large inter-frame intervals in fast motion, leading to the loss of crucial dynamic information; simply increasing the frame rate incurs significant storage bandwidth costs and requires strong lighting conditions, limiting practical applications. While event cameras can capture fast-moving edges with microsecond-level temporal resolution, their output only encodes brightness changes, lacking texture details and exhibiting sluggish response to static areas.

[0006] Existing event-based 3D reconstruction methods are mostly limited to static or quasi-static scenes. Even when some methods achieve dynamic reconstruction, the lack of texture information makes it difficult to obtain 3D point trajectories that combine high temporal resolution and texture fidelity. Therefore, how to effectively integrate the texture fidelity of RGB images with the high temporal resolution of event data to achieve high frame rate and high-precision 3D point trajectory reconstruction in high-speed motion scenes has become an urgent technical problem to be solved.

[0007] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a method and system for high-speed motion 3D point tracking and trajectory reconstruction, which addresses the above-mentioned deficiencies of the prior art and aims to solve the problem that the prior art is unable to achieve high frame rate and high precision 3D point trajectory reconstruction in high-speed motion scenarios.

[0009] The technical solution adopted by this invention to solve the problem is as follows: In a first aspect, embodiments of the present invention provide a method for high-speed motion three-dimensional point tracking and trajectory reconstruction, the method comprising: Simultaneous acquisition of RGB image sequences and event streams using a binocular multimodal observation device; The RGB image sequence and the event stream are time-aligned; RGB feature maps are extracted based on the RGB image sequence; and event feature maps are extracted based on the event stream. The RGB feature map and the event feature map are mapped to a three-dimensional space; a three-dimensional feature point cloud is supplemented by a linear interpolation algorithm; and the spatial distribution of the supplemented three-dimensional feature point cloud is balanced by an anchor point sampling algorithm to obtain a fused three-dimensional feature point cloud. High-speed 3D point trajectory data is generated based on the fused 3D feature point cloud using a time-correlation transformer.

[0010] In one implementation, the binocular multimodal observation device includes two hybrid cameras; each hybrid camera includes an event camera and an optical RGB camera, and the incident light is split into two paths by a beam splitter, one path being connected to the event camera and the other path being connected to the optical RGB camera; the data from the two hybrid cameras are synchronized and encoded by hardware time, and then decoded and stored in the host workstation.

[0011] In one implementation, the steps of time-aligning the RGB image sequence with the event stream, extracting RGB feature maps based on the RGB image sequence, and extracting event feature maps based on the event stream include: The event stream during the time series containing the target time is evenly distributed to multiple consecutive non-overlapping time intervals to construct an event voxel grid; An optical feature extractor based on a convolutional neural network is used to extract dense appearance features from the RGB image sequence to obtain an RGB feature map. An event feature map is obtained by extracting a time-dependent fine-grained two-dimensional feature map from the event voxel grid using an event feature extractor built based on a hybrid deep learning network. The event feature map and the RGB feature map are matched in dimension by projection transformation.

[0012] In one implementation, the steps for supplementing the 3D feature point cloud using a linear interpolation algorithm include: For the selected RGB camera viewpoint, two adjacent low frame rate moments before and after the target moment are used as reference moments; the low frame rate moment is the moment when the frame rate is lower than a preset threshold. The three-dimensional feature point cloud at the target time is supplemented by a linear interpolation algorithm based on the three-dimensional feature point clouds at the two reference times.

[0013] In one implementation, the anchor point sampling algorithm is a farthest point sampling algorithm; the step of performing spatial distribution equalization on the supplemented 3D feature point cloud using the anchor point sampling algorithm to obtain the fused 3D feature point cloud includes: The RGB 3D feature point cloud from each viewpoint is aggregated with the event 3D feature point cloud to obtain the aggregated point cloud. The algorithm of sampling the farthest point is used to select iterative points according to the principle of maximizing the minimum distance between sampling points, and the aggregated point cloud is iteratively sampled to obtain a fused three-dimensional feature point cloud.

[0014] In one implementation, the step of generating high-speed 3D point trajectory data based on the fused 3D feature point cloud using a time-correlation transformer includes: At least one point is selected as a query point in the fused three-dimensional feature point cloud at the target time. For each query point, several nearest neighbor points corresponding to the query point are determined by the fused 3D feature point cloud of adjacent frames; Relational terms are constructed based on each nearest neighbor, and the relational terms are input into a time-related transformer; The time-related transformer predicts the continuous three-dimensional spatial coordinates of the query point based on the relational terms. High-speed three-dimensional point trajectory data is generated based on the continuous three-dimensional spatial coordinates of the query point.

[0015] In one implementation, the time-dependent transformer is initialized with pre-trained weights and its parameters are fine-tuned using a low-rank adaptive fine-tuning method.

[0016] Secondly, embodiments of the present invention also provide a high-speed motion three-dimensional point tracking and trajectory reconstruction device, the device comprising: A binocular multimodal observation device for simultaneously acquiring RGB image sequences and event streams; A dual-modal two-dimensional feature extractor is used to time-align the RGB image sequence with the event stream, extract RGB feature maps based on the RGB image sequence, and extract event feature maps based on the event stream; The heterogeneous 3D feature fusion module is used to map the RGB feature map and the event feature map to a 3D space; supplement the 3D feature point cloud through a linear interpolation algorithm; and perform spatial distribution equalization on the supplemented 3D feature point cloud through an anchor point sampling algorithm to obtain a fused 3D feature point cloud. The trajectory reconstruction module is used to generate high-speed three-dimensional point trajectory data based on the fused three-dimensional feature point cloud through a time-correlation transformer.

[0017] Thirdly, embodiments of the present invention also provide a terminal, the terminal including a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the high-speed motion three-dimensional point tracking and trajectory reconstruction method as described above; the processor is used to execute the programs.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of instructions adapted to be loaded and executed by a processor to implement the steps of the high-speed motion three-dimensional point tracking and trajectory reconstruction method as described above.

[0019] The beneficial effects of this invention are as follows: This invention employs a binocular multimodal observation device to acquire multimodal observation information for high-speed motion. Features are extracted from both modalities and then heterogeneous 3D features are fused. Finally, a time-correlation transformer is used to predict the 3D trajectory of high-speed moving points. This invention effectively combines the texture fidelity of RGB images with the high temporal resolution of event data, enabling high frame rate and high-precision 3D point trajectory reconstruction in high-speed motion scenes. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the high-speed motion three-dimensional point tracking and trajectory reconstruction method provided in this embodiment of the invention.

[0022] Figure 2 This is a schematic diagram of a hybrid camera provided in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of heterogeneous three-dimensional feature fusion provided in an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram comparing the effects of the present invention with those of the prior art, provided by an embodiment of the present invention.

[0025] Figure 5 This is a schematic diagram of the technical architecture of the high-speed motion three-dimensional point tracking and trajectory reconstruction method provided in the embodiments of the present invention.

[0026] Figure 6 This is a schematic diagram of the module of the high-speed motion three-dimensional point tracking and trajectory reconstruction device provided in the embodiment of the present invention.

[0027] Figure 7 This is a schematic diagram of the terminal provided in the embodiment of the present invention. Detailed Implementation

[0028] This invention discloses a method and system for high-speed motion three-dimensional point tracking and trajectory reconstruction. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0029] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0030] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0031] To address the aforementioned deficiencies in existing technologies, this invention provides a method for high-speed motion 3D point tracking and trajectory reconstruction. The method includes: simultaneously acquiring RGB image sequences and event streams using a binocular multimodal observation device; temporally aligning the RGB image sequences and event streams; extracting RGB feature maps based on the RGB image sequences and event feature maps based on the event streams; mapping the RGB feature maps and event feature maps to a 3D space; supplementing a 3D feature point cloud using a linear interpolation algorithm; performing spatial distribution equalization on the supplemented 3D feature point cloud using an anchor point sampling algorithm to obtain a fused 3D feature point cloud; and generating high-speed 3D point trajectory data based on the fused 3D feature point cloud using a time-correlation transformer. This invention employs a binocular multimodal observation device to acquire multimodal observation information oriented towards high speed, extracts features from both modalities separately, performs heterogeneous 3D feature fusion, and finally uses a time-correlation transformer to predict the 3D trajectory of high-speed motion points. This invention effectively integrates the texture fidelity of RGB images with the high temporal resolution of event data, enabling high frame rate and high-precision 3D point trajectory reconstruction in high-speed motion scenarios.

[0032] like Figure 1 As shown, the method includes: Step S100: Synchronously acquire RGB image sequences and event streams using a binocular multimodal observation device.

[0033] Furthermore, the binocular multimodal observation device includes two hybrid cameras; each hybrid camera includes an event camera and an optical RGB camera, and the incident light is split into two paths by a beam splitter, one path is connected to the event camera and the other path is connected to the optical RGB camera; the data from the two hybrid cameras are synchronized and encoded by hardware time, and then decoded and stored in the host workstation.

[0034] Specifically, this embodiment constructs a binocular multimodal observation device for high-speed motion scenes. By combining it with a multi-view, multimodal spatiotemporally synchronized event-RGB fusion data acquisition mechanism, it can simultaneously acquire multi-view high-resolution stereo information and high frame rate texture dynamic data.

[0035] The binocular multimodal observation device employs an event-guided cube-shaped multi-view stereo capture architecture, consisting of two sets of hybrid cameras forming the binocular observation system. For example... Figure 2 As shown, each hybrid camera includes an event camera and an optical RGB camera. Each hybrid camera uses a beam splitter to separate the incident light into two paths, which are then guided to the event camera and the optical RGB camera respectively under spatial consistency conditions, thereby achieving synchronous acquisition of continuous pulse event signal streams and discrete RGB image frames.

[0036] In one implementation, the hybrid camera prototype uses an optical RGB camera operating at 30fps with a resolution of 1080×720; the event camera (EVK4 type) provides microsecond-level temporal resolution and 1080×720 spatial resolution. Data acquired by the two hybrid cameras is transmitted via fiber optic cable to an onboard processing chip for hardware time synchronization and unified encoding, followed by decoding and storage by the host workstation. During data acquisition, each camera channel maintains strict time synchronization to ensure consistent temporal representation of motion trajectories from different perspectives.

[0037] In practical applications, the data set construction process based on a binocular multimodal observation device includes: obtaining extrinsic and intrinsic parameters of a fixed multi-camera system through spatiotemporal calibration; generating an initial point cloud using a depth estimation network as the query 3D point; obtaining depth maps from optical RGB cameras using the same depth estimation network, and generating depth maps from event cameras using temporal interpolation of RGB depth maps. To compensate for the shortcomings of real-world testing platforms in terms of training and quantitative evaluation benchmarks due to the lack of depth cameras, this embodiment further constructs a virtual binocular multimodal observation device in a physical simulation environment, synthesizing a large-scale dataset containing complete point trajectory real values; simultaneously, it synthesizes and modifies publicly available datasets, obtaining low-frame-rate RGB sequences and microsecond-level temporal resolution event streams as inputs through frame rate downsampling, motion blur overlay, and event transformation, and using the original 3D trajectory as a high-frame-rate real trajectory reference, providing effective data support for high-speed 3D point tracking.

[0038] For example, to achieve high-speed 3D point tracking and acquire accurate motion trajectories at high frame rates, this embodiment employs a multi-view Event-RGB hybrid stereo vision capture device and constructs a point tracking method based on discrete RGB frames and continuous event streams. This tracking problem can be formally expressed as: ; In the formula, This represents RGB frames captured from n optical cameras; Indicates from The event streams captured by each event camera; Indicates the target time Predicted trajectory of three-dimensional points.

[0039] Specifically, for the nth optical camera, the RGB frames it captures It can be described as follows: ; In the formula, Indicates the first Frame time RGB image at that location, This represents the total number of frames captured by the optical camera (depending on the shooting frame rate, e.g., 30fps).

[0040] For the An event camera that captures a continuous stream of events. This can be expressed as: ; In the formula, This indicates the j-th capture event of the event camera, which records the intensity change of a specific pixel; () represents the two-dimensional coordinates of the pixel; Indicates the event polarity (i.e., whether the intensity of the corresponding pixel increases or decreases); Indicates the target time, i.e., the time when the event occurs; Indicates the total number of events.

[0041] Regarding the construction of the 3D point tracking dataset, this embodiment selects small, fast-moving, and representative experimental objects, including five typical scenes such as a shaking cube, a rotating plane, and a swinging toy. During data acquisition, the capture device remains stationary, and the object moves within a defined area centered on the device. Each scene record provides two perspectives: two RGB perspectives containing discrete frame images acquired at a rate of 30 frames per second, and two event perspectives storing event tuples with microsecond-level temporal resolution. , , , The dataset construction process specifically includes: first, obtaining the extrinsic and intrinsic parameters of a fixed camera through spatiotemporal calibration; then, obtaining the initial point cloud data source through a depth estimation network (such as Depth Anything V3) and using it as the query 3D points; the depth map data of the RGB camera is also obtained through the depth estimation network, while the depth map of the event camera is obtained from the depth map of the RGB camera through frame interpolation.

[0042] To obtain a large-scale dataset containing complete real-world point trajectory data, this embodiment, referencing the research methodology of Multi-View Point Tracking (MVTracker), constructs a virtual device within the Kubric (an open-source data generation tool) physics simulator. The virtual device includes two spatiotemporally synchronized low-frame-rate optical cameras and an event camera. Multiple objects undergo rapid free-fall motion from a fixed height at the scene center, with all cameras simultaneously recording data. This embodiment synthesizes a total of 100 scene datasets, of which 70 are used for model training and 30 for model testing. Camera parameters, 3D query points, depth maps, and real-world trajectory data at high frame rates are directly extracted from the Kubric simulator, thus providing ground truth-annotated supervisory signals for model training and quantitative evaluation.

[0043] Furthermore, this embodiment also constructs a real-world benchmark by synthesizing and modifying the Panoptic public dataset. Specifically, two suitable viewpoints are selected from the Panoptic dataset, and the image frames from these two viewpoints are first converted into continuous event streams using a v2e tool. Next, the image frame sequence is processed by frame extraction and motion blur is applied to simulate the acquisition characteristics of low-frame-rate discrete optical data. The resulting two sets of low-frame-rate RGB sequences and two sets of microsecond-level temporal resolution event streams constitute the input data, while the original 3D trajectory from the Panoptic dataset is used as a substitute for a high-frame-rate real trajectory for quantitative error evaluation and performance comparison of the model's prediction results. Through the complementary construction of the virtual synthetic data and the real-world benchmark data, the problem of insufficient training data in high-speed 3D point tracking tasks is effectively solved.

[0044] Step S200: Time-align the RGB image sequence with the event stream, extract RGB feature maps based on the RGB image sequence, and extract event feature maps based on the event stream.

[0045] Specifically, since optical cameras generate discrete RGB image frames at a low frame rate, while event cameras record continuous asynchronous event streams, there are differences in data format and temporal density. Therefore, it is necessary to unify the processing of the two modalities. In practical applications, the RGB image sequences and event streams are first time-aligned to establish a correspondence between them under the same time reference. Based on this, two-dimensional feature extraction is performed for each modality. For RGB image sequences, corresponding RGB feature maps (also known as RGB two-dimensional feature maps) are generated through convolutional neural networks. To improve the fine-grainedness of feature representation, feature maps can be further calculated at multiple different scales to retain rich texture and structural information. For event streams, due to the strong temporal order dependence and low texture information at a single time step, the event stream can be transformed into a time-dependent fine-grained event feature map (also known as an event two-dimensional feature map) by combining event voxel grids with a hybrid deep learning network (i.e., a CNN-LSTM network, composed of convolutional layers and a long short-term memory network).

[0046] In one implementation, the steps of time-aligning the RGB image sequence with the event stream, extracting RGB feature maps based on the RGB image sequence, and extracting event feature maps based on the event stream include: The event stream during the time series containing the target time is evenly distributed to multiple consecutive non-overlapping time intervals to construct an event voxel grid; An optical feature extractor based on a convolutional neural network is used to extract dense appearance features from the RGB image sequence to obtain an RGB feature map. An event feature map is obtained by extracting a time-dependent fine-grained two-dimensional feature map from the event voxel grid using an event feature extractor built based on a hybrid deep learning network. The event feature map and the RGB feature map are matched in dimension by projection transformation.

[0047] Specifically, this embodiment constructs a bimodal two-dimensional feature extractor to extract complementary three-dimensional motion dynamics features from data of two modalities. First, the RGB image sequence and event stream are temporally aligned to establish a frame-event correspondence under the same time reference. Based on this, a suitable feature extraction network is designed for each modality. For the event stream, since the event data is asynchronous and sparse, this embodiment evenly distributes the event stream during the time series containing the target time to multiple consecutive non-overlapping time intervals, constructing an event voxel grid, thereby normalizing the asynchronous events into a spatiotemporal tensor form. Subsequently, the event voxel grid is input into a hybrid deep learning network to extract time-dependent fine-grained two-dimensional feature maps, obtaining event feature maps. The hybrid deep learning network (i.e., CNN-LSTM network) integrates convolutional neural networks and long short-term memory networks, where the convolutional neural network is responsible for capturing spatial local patterns, and the long short-term memory network is responsible for transmitting motion state information along the time dimension, thereby effectively extracting high-frequency motion features contained in the event stream. For the RGB image sequence, this embodiment inputs the RGB image sequence into a convolutional neural network to extract dense appearance features, obtaining RGB feature maps. Convolutional neural networks extract dense feature maps containing texture, edge, and structural information at multiple scales through multi-layer convolution and pooling operations, in order to preserve the appearance and spatial layout information of objects.

[0048] Because the feature maps output by the two feature extraction networks differ in channel dimension and spatial resolution, they cannot be directly correlated heterogeneously. This embodiment further uses projection transformation to perform dimensionality matching between the event feature map and the RGB feature map, ensuring consistency in feature dimension. Thus, the bimodal 2D feature extractor outputs time-aligned and dimensionally unified RGB and event feature maps, providing an input foundation for subsequent upscaling of 2D features to 3D space and heterogeneous fusion.

[0049] For example, regarding time series The asynchronous event stream during this period is converted into an event voxel grid for subsequent processing and time alignment. Events within the sliding window (also known as the time series window) corresponding to this time series are evenly distributed into B (B=5) consecutive and non-overlapping time intervals, with each event simultaneously affecting its two adjacent intervals.

[0050] For discrete RGB frames, dense appearance features are extracted using a convolutional neural network. , This represents a convolutional neural network. To extract more detailed features, feature maps can be further computed at four different scales to obtain RGB feature maps.

[0051] For a continuous stream of events, a CNN-LSTM network is pre-built to extract time-dependent fine-grained 2D feature maps from the event camera voxel grid. , This represents a CNN-LSTM network. This CNN-LSTM network consists of three convolutional layers and a long short-term memory network, and projects event features to match the RGB feature dimensions.

[0052] Step S300: Map the RGB feature map and the event feature map to a three-dimensional space; supplement the three-dimensional feature point cloud using a linear interpolation algorithm; perform spatial distribution equalization on the supplemented three-dimensional feature point cloud using an anchor point sampling algorithm to obtain a fused three-dimensional feature point cloud.

[0053] Specifically, after extracting multi-view and multi-modal 2D feature maps, these features need to be mapped to a unified 3D space to obtain a consistent multi-view representation for spatial tracking. To this end, this scheme constructs a heterogeneous 3D feature fusion mechanism, aiming to extract complementary 3D motion dynamics features from the RGB modality and the event modality, and generate a high-frame-rate, accurate 3D spatiotemporal representation through fusion to compensate for the deficiencies of a single modality in the temporal or spatial domains. First, a 3D feature enhancement method is used to enhance each effective pixel on the 2D plane in 3D space using depth information and camera parameters. The 2D feature points extracted from all RGB cameras and the event camera are enhanced to a unified world coordinate system, forming RGB 3D feature point clouds and event 3D feature point clouds.

[0054] However, as Figure 3 As shown, when fusing 3D features at high frame rates, simply stitching together point clouds from all perspectives in each high frame rate frame leads to two key problems: first, temporal non-smoothness, where the fused 3D features degrade drastically at times lacking low frame rate RGB frames; and second, spatial imbalance, where simple merging causes excessive clustering of feature points in some local areas while other areas are undersampled, disrupting the spatial uniformity of the point cloud. To address the temporal non-smoothness issue, this embodiment uses linear interpolation for smoothing. Interpolation ensures complete RGB 3D feature point clouds at all high frame rate moments, eliminating feature degradation caused by missing RGB frames. To address the spatial imbalance issue, this embodiment employs a balancing strategy of fusion followed by anchor point sampling, maintaining the original point cloud geometry while achieving spatial distribution balance. After anchor point sampling, a spatially uniform fused 3D feature point cloud is obtained.

[0055] For example, each effective pixel on a two-dimensional plane ( 3D spatial augmentation can be achieved using depth information and camera parameters: ; In the formula, and They represent the first Each camera at frame time External and internal references at the time This represents a depth map. It can upscale 2D points to 3D points from all RGB cameras and event cameras. Each three-dimensional point Each has its corresponding two-dimensional feature map: ; ; In the formula, Indicates from the first The RGB camera in the 3D feature point cloud extracted from frames; Indicates from the first The event camera at the 3D feature point cloud extracted from frames; express A very short time interval in the vicinity; It represents the set of valid pixels on a two-dimensional plane.

[0056] To address the aforementioned temporal non-smoothness issue, this embodiment employs linear interpolation to achieve smoothness: given that high-speed motion trajectories typically exhibit strong temporal continuity, the evolution process in the RGB feature space is approximated as linear, and the missing high-frame-rate 3D feature point cloud is recovered through linear interpolation.

[0057] To address the aforementioned spatial imbalance issue, this embodiment employs a balancing strategy of first fusing and then anchor-point sampling: First, the two modes are aggregated, and then an anchor-point-based sampling method is applied. Anchor-point sampling achieves a more balanced spatial distribution while preserving the original point cloud geometry and spatial distribution characteristics.

[0058] In one implementation, the steps for supplementing the 3D feature point cloud using a linear interpolation algorithm include: For the selected RGB camera viewpoint, two adjacent low frame rate moments before and after the target moment are used as reference moments; the low frame rate moment is the moment when the frame rate is lower than a preset threshold. The three-dimensional feature point cloud at the target time is supplemented by a linear interpolation algorithm based on the three-dimensional feature point clouds at the two reference times.

[0059] Specifically, to address the aforementioned temporal non-smoothness issue, the temporal evolution of RGB features is approximated as a linear process: for a given RGB camera viewpoint, let... and For two consecutive timestamps at a low frame rate (e.g., 30fps); ∈ ( , ), This is the target timestamp (i.e., target time) at a higher frame rate (e.g., 150fps). Time points are supplemented using the following linear weighting method. 3D feature point cloud at the location: ; ; In the formula, This represents the linear interpolation coefficients.

[0060] Through the above interpolation process, complete RGB three-dimensional feature point clouds can be obtained at each moment of high frame rate, effectively eliminating the feature degradation phenomenon caused by missing RGB frames.

[0061] In one implementation, the anchor point sampling algorithm is a farthest point sampling algorithm; the step of performing spatial distribution equalization on the supplemented 3D feature point cloud using the anchor point sampling algorithm to obtain the fused 3D feature point cloud includes: The RGB 3D feature point cloud from each viewpoint is aggregated with the event 3D feature point cloud to obtain the aggregated point cloud. The algorithm of sampling the farthest point is used to select iterative points according to the principle of maximizing the minimum distance between sampling points, and the aggregated point cloud is iteratively sampled to obtain a fused three-dimensional feature point cloud.

[0062] Specifically, after linear interpolation supplementation of the RGB 3D feature point clouds and the event 3D feature point clouds from various perspectives, the two types of point clouds differ in spatial distribution: the RGB feature point cloud is relatively sparse, while the event feature point cloud has high density but uneven distribution. Directly merging the two would lead to excessive clustering of feature points in local areas and insufficient sampling in other areas, disrupting the spatial uniformity of the point cloud and thus affecting subsequent prediction accuracy. Therefore, this embodiment adopts a strategy of aggregation followed by equalization: first, the RGB 3D feature point clouds and the event 3D feature point clouds from various perspectives are aggregated to obtain an aggregated point cloud. For the target timestamp... From RGB cameras and The entire 3D feature point set is acquired from each event camera to generate a fused 3D feature point cloud. The calculation formula is as follows: ; In the formula, This represents the fused 3D feature points obtained from n RGB cameras and m event cameras; FPS represents the farthest point sampling algorithm used for anchor point extraction. This represents the set of all 3D feature points from n RGB cameras at the target time. This represents the set of all 3D feature points of m event cameras at the target time.

[0063] The farthest point sampling algorithm iteratively selects points based on maximizing the minimum distance between sampling points and iteratively samples the aggregated point cloud. Similar to the processing method in PointNet++ (a deep learning model specifically for handling 3D point cloud data), farthest point sampling selects points iteratively by maximizing the minimum distance between sampling points. That is, in each iteration, the point with the largest minimum distance from the unselected points to the selected point set is chosen, until a preset number of samples is reached. This process effectively eliminates the problems of local over-density and sparseness caused by simple superposition of multimodal and multi-view point clouds while maintaining the original point cloud geometry, achieving a balanced spatial distribution of the 3D feature point cloud. After anchor point sampling, the fused 3D feature point cloud is obtained. The point cloud retains the geometric topology and multimodal feature information of the original point cloud with uniformly distributed spatial anchor points, providing a well-structured three-dimensional feature input for subsequent spatiotemporal nearest neighbor association and trajectory prediction.

[0064] Step S400: Generate high-speed 3D point trajectory data based on the fused 3D feature point cloud using a time-correlation transformer.

[0065] Furthermore, the step of generating high-speed 3D point trajectory data based on the fused 3D feature point cloud using a time-correlated transformer includes: At least one point is selected as a query point in the fused three-dimensional feature point cloud at the target time. For each query point, several nearest neighbor points corresponding to the query point are determined by the fused 3D feature point cloud of adjacent frames; Relational terms are constructed based on each nearest neighbor, and the relational terms are input into a time-related transformer; The time-related transformer predicts the continuous three-dimensional spatial coordinates of the query point based on the relational terms. High-speed three-dimensional point trajectory data is generated based on the continuous three-dimensional spatial coordinates of the query point.

[0066] Furthermore, the time-correlated transformer is initialized with pre-trained weights and its parameters are fine-tuned using a low-rank adaptive fine-tuning method.

[0067] Specifically, after obtaining a high-frame-rate, spatially balanced fused 3D feature point cloud through heterogeneous 3D feature fusion, it is necessary to establish a temporally continuous correlation between the high-frame-rate 3D spatiotemporal representation and each query point to guide the model in learning generalizable 3D high-speed motion trajectories. To this end, this embodiment constructs a temporal correlation transformer capable of recovering high-density, high-frame-rate 3D point trajectories. Unlike using spatial correlation modules (such as triplane projection or k-nearest neighbor aggregation) to capture pure spatial correlation, this embodiment integrates temporal correlation to construct a spatiotemporal joint nearest neighbor correlation model, thus obtaining the temporal correlation transformer.

[0068] In practical applications, at least one point is selected as a query point from the fused 3D feature point cloud at the target time. The query point is the target 3D point to be tracked, and its initial position can be determined using the aforementioned initial point cloud data source. For each query point, several nearest neighbor points (i.e., spatial nearest neighbor feature points) are determined using the fused 3D feature point clouds of adjacent frames. For the target timestamp... Each query point In this embodiment, K nearest neighbor points (also called nearest neighbors) are retrieved from the fused 3D feature point clouds of the previous frame, the current frame, and the next frame, respectively, to construct a spatiotemporal joint nearest neighbor set, the expression of which is: ; In the formula, Indicates the number of nearest neighbors; This indicates the fusion of 3D feature point clouds. Current position estimate Surrounding Nearest neighbor points. These represent the spatial coordinates of the nearest neighbor and their corresponding fusion features, respectively. express Three-dimensional feature points are associated with their nearest neighbors. At high frame rates, motion exhibits "slow-motion" characteristics, meaning that the position in each frame is highly correlated with the positions in its preceding and following frames. By modeling this temporal continuity, stronger representational capabilities can be obtained while maintaining continuity.

[0069] Subsequently, relational lexical units are constructed based on each nearest neighbor. , The encoder is represented by a relational lexicon that is iteratively passed to a temporal correlation transformer to obtain complete high-frame-rate 3D trajectory data. This approach can fully utilize large-scale datasets to improve generalization ability and effectively reduce computational overhead.

[0070] Furthermore, given that the time-dependent transformer is a Transformer model, and considering the comprehensive factors of training cost, dataset preparation, and computational overhead, this embodiment can use MVTracker pre-trained Transformer weights to initialize the time-dependent transformer, and then use the low-rank adaptation (LoRA) fine-tuning method to efficiently fine-tune the parameters on a self-built synthetic dataset.

[0071] To facilitate understanding of the technical solution of the present invention, such as Figure 4 As shown in the figure, this invention illustrates the overall framework of its method. The invention first uses a binocular multimodal observation device to simultaneously acquire data from a high-speed moving target, obtaining RGB image sequences (RGB1, RGB2) and event stream data (DVS1, DVS2) from two perspectives. The RGB image sequences are input to an optical feature extractor, where multi-scale dense appearance features are extracted via a convolutional neural network to generate corresponding RGB feature maps. The event stream data are input to an event feature extractor, where time-dependent fine-grained features are extracted via voxel meshing and a CNN-LSTM hybrid network to generate corresponding event feature maps. Subsequently, the RGB feature maps and event feature maps are jointly input to a heterogeneous 3D feature fusion module, which elevates the 2D features to 3D space. Linear interpolation is used to supplement missing time-stamp features, and farthest-point sampling is used to balance the spatial distribution, resulting in a fused 3D feature point cloud. Finally, the 3D feature point cloud is fused into the temporal correlation transformer. By combining the previous frame, the current frame, and the next frame for spatiotemporal nearest neighbor correlation and iterative decoding, a high frame rate and high density continuous high-speed 3D point trajectory is recovered, thereby realizing a high-speed tracking process from multimodal input to 3D motion trajectory output.

[0072] In summary, the technical effects of the present invention include, but are not limited to: 1. High-precision 3D perception in high-speed scenes is achieved through multimodal complementary observation: This invention uses a binocular multimodal observation device to heterogeneously combine a high-resolution but low-frame-rate RGB camera with a microsecond-level time resolution event camera, providing a complete multimodal observation data foundation for high-speed 3D point tracking.

[0073] 2. By extracting features from two modalities and aligning them with time sequences, complementary motion dynamics information is fully explored: The two-dimensional feature extractor constructed in this invention uses a multi-scale convolutional network to extract dense appearance features for the RGB modality and a CNN-LSTM hybrid network to extract time-dependent features for the event modality. Feature dimension matching is achieved through projection transformation.

[0074] 3. By employing a fusion strategy that combines linear interpolation and anchor point sampling, the inherent defects in the spatiotemporal domain are simultaneously eliminated: In the fusion of heterogeneous 3D features, this invention utilizes linear interpolation to recover the missing RGB 3D features at high frame rates, eliminating the temporal non-smooth degradation caused by insufficient RGB frame rates; on the other hand, it employs farthest point sampling to spatially equalize the aggregated point cloud, solving the problem of local over-density and sparse unevenness caused by simple merging of multi-view multimodal point clouds.

[0075] 4. Improved prediction accuracy and continuity of high-speed motion trajectory through spatiotemporal joint nearest neighbor modeling: The temporal correlation transformer constructed in this invention constructs a spatiotemporal nearest neighbor set for each query point by combining the fused three-dimensional feature point cloud of the previous frame, the current frame and the next frame, making full use of the strong coherence of high-speed motion in the temporal domain, and effectively improving the accuracy of trajectory prediction at high frame rates.

[0076] 5. By combining pre-training initialization with LoRA fine-tuning, a good balance between generalization ability and training efficiency is achieved: This invention uses large-scale pre-trained Transformer weights as initialization parameters and performs efficient parameter fine-tuning on a self-built synthetic dataset through the low-rank adaptation (LoRA) method, achieving a good balance between high-precision trajectory prediction and lightweight deployment.

[0077] This invention compares itself with existing technologies on both synthetic and real datasets. The datasets contain numerous high-speed motion scenarios, such as objects rapidly falling from heights, basketball passing, small ball throwing, and small objects moving quickly indoors. Figure 5 As shown in the figure, the qualitative performance analysis results show that the predicted trajectory of the present invention has the highest degree of agreement with the actual motion trajectory of the object.

[0078] Based on the above embodiments, the present invention also provides a high-speed motion three-dimensional point tracking and trajectory reconstruction device, such as... Figure 6 As shown, the device includes: Binocular multimodal observation device 01, used for synchronous acquisition of RGB image sequences and event streams; A dual-modal two-dimensional feature extractor 02 is used to perform time alignment between the RGB image sequence and the event stream, extract RGB feature maps based on the RGB image sequence, and extract event feature maps based on the event stream; The heterogeneous 3D feature fusion module 03 is used to map the RGB feature map and the event feature map to a 3D space; supplement the 3D feature point cloud through a linear interpolation algorithm; and perform spatial distribution balancing on the supplemented 3D feature point cloud through an anchor point sampling algorithm to obtain a fused 3D feature point cloud. The trajectory reconstruction module 04 is used to generate high-speed three-dimensional point trajectory data based on the fused three-dimensional feature point cloud through a time-correlation transformer.

[0079] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 7 As shown, the terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a high-speed motion 3D point tracking and trajectory reconstruction method. The display screen can be an LCD screen or an e-ink screen.

[0080] Those skilled in the art will understand that Figure 7 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0081] In one implementation, the terminal's memory stores one or more programs, and these programs are configured to be executed by one or more processors, and the programs contain instructions for performing a high-speed motion three-dimensional point tracking and trajectory reconstruction method.

[0082] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0083] In summary, this invention discloses a method and system for high-speed motion 3D point tracking and trajectory reconstruction. The method includes: simultaneously acquiring RGB image sequences and event streams using a binocular multimodal observation device; temporally aligning the RGB image sequences and event streams; extracting RGB feature maps based on the RGB image sequences and event feature maps based on the event streams; mapping the RGB feature maps and event feature maps to a 3D space; supplementing a 3D feature point cloud using a linear interpolation algorithm; performing spatial distribution equalization on the supplemented 3D feature point cloud using an anchor point sampling algorithm to obtain a fused 3D feature point cloud; and generating high-speed 3D point trajectory data based on the fused 3D feature point cloud using a time-correlation transformer. This invention uses a binocular multimodal observation device to acquire multimodal observation information oriented towards high speed, extracts features from both modalities separately, performs heterogeneous 3D feature fusion, and finally uses a time-correlation transformer to predict the 3D trajectory of high-speed motion. This invention effectively integrates the texture fidelity of RGB images with the high temporal resolution of event data, enabling high frame rate and high-precision 3D point trajectory reconstruction in high-speed motion scenarios.

[0084] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for high-speed motion three-dimensional point tracking and trajectory reconstruction, characterized in that, The method includes: RGB image sequences and event streams are simultaneously acquired using a binocular multimodal observation device. The binocular multimodal observation device includes two hybrid cameras. Each hybrid camera includes an event camera and an optical RGB camera. The incident light is split into two paths by a beam splitter, one path connecting to the event camera and the other path connecting to the optical RGB camera. The data from the two hybrid cameras are synchronized and encoded by hardware, then decoded and stored on the host workstation. The RGB image sequence and the event stream are time-aligned; RGB feature maps are extracted based on the RGB image sequence; and event feature maps are extracted based on the event stream. The RGB feature map and the event feature map are mapped to a three-dimensional space; the three-dimensional feature point cloud is supplemented by a linear interpolation algorithm. The spatial distribution of the supplemented 3D feature point cloud is balanced by an anchor point sampling algorithm to obtain a fused 3D feature point cloud. The anchor point sampling algorithm is a farthest point sampling algorithm, which includes: aggregating the RGB 3D feature point cloud and the event 3D feature point cloud from each viewpoint to obtain an aggregated point cloud; and iteratively selecting points based on the principle of maximizing the minimum distance between sampling points using the farthest point sampling algorithm, and iteratively sampling the aggregated point cloud to obtain the fused 3D feature point cloud. High-speed 3D point trajectory data is generated based on the fused 3D feature point cloud using a time-correlation transformer.

2. The high-speed motion three-dimensional point tracking and trajectory reconstruction method according to claim 1, characterized in that, The steps of temporally aligning the RGB image sequence with the event stream, extracting RGB feature maps based on the RGB image sequence, and extracting event feature maps based on the event stream include: The event stream during the time series containing the target time is evenly distributed to multiple consecutive non-overlapping time intervals to construct an event voxel grid; An optical feature extractor based on a convolutional neural network is used to extract dense appearance features from the RGB image sequence to obtain an RGB feature map. An event feature map is obtained by extracting a time-dependent fine-grained two-dimensional feature map from the event voxel grid using an event feature extractor built based on a hybrid deep learning network. The event feature map and the RGB feature map are matched in dimension by projection transformation.

3. The high-speed motion three-dimensional point tracking and trajectory reconstruction method according to claim 2, characterized in that, The steps for supplementing a 3D feature point cloud using a linear interpolation algorithm include: For the selected RGB camera viewpoint, two adjacent low frame rate moments before and after the target moment are used as reference moments; the low frame rate moment is the moment when the frame rate is lower than a preset threshold. The three-dimensional feature point cloud at the target time is supplemented by a linear interpolation algorithm based on the three-dimensional feature point clouds at the two reference times.

4. The high-speed motion three-dimensional point tracking and trajectory reconstruction method according to claim 2, characterized in that, The steps of generating high-speed 3D point trajectory data based on the fused 3D feature point cloud using a time-correlation transformer include: At least one point is selected as a query point in the fused three-dimensional feature point cloud at the target time. For each query point, several nearest neighbor points corresponding to the query point are determined by the fused 3D feature point cloud of adjacent frames; Relational terms are constructed based on each nearest neighbor, and the relational terms are input into a time-related transformer; The time-related transformer predicts the continuous three-dimensional spatial coordinates of the query point based on the relational terms. High-speed three-dimensional point trajectory data is generated based on the continuous three-dimensional spatial coordinates of the query point.

5. The high-speed motion three-dimensional point tracking and trajectory reconstruction method according to claim 4, characterized in that, The time-correlated transformer is initialized with pre-trained weights and its parameters are fine-tuned using a low-rank adaptive fine-tuning method.

6. A high-speed motion three-dimensional point tracking and trajectory reconstruction device, characterized in that, The device includes: A binocular multimodal observation device is used to simultaneously acquire RGB image sequences and event streams. The binocular multimodal observation device includes two hybrid cameras. Each hybrid camera includes an event camera and an optical RGB camera. The incident light is split into two paths by a beam splitter, one path connected to the event camera and the other path connected to the optical RGB camera. The data from the two hybrid cameras are synchronized and encoded by hardware, then decoded and stored in the host workstation. A dual-modal two-dimensional feature extractor is used to time-align the RGB image sequence with the event stream, extract RGB feature maps based on the RGB image sequence, and extract event feature maps based on the event stream; A heterogeneous 3D feature fusion module is used to map the RGB feature map and the event feature map to a 3D space; supplement the 3D feature point cloud using a linear interpolation algorithm; and perform spatial distribution equalization on the supplemented 3D feature point cloud using an anchor point sampling algorithm to obtain a fused 3D feature point cloud. The anchor point sampling algorithm is a farthest point sampling algorithm, which includes: aggregating the RGB 3D feature point cloud and the event 3D feature point cloud from each viewpoint to obtain an aggregated point cloud; and using the farthest point sampling algorithm, iteratively selecting points based on the principle of maximizing the minimum distance between sampling points, and iteratively sampling the aggregated point cloud to obtain the fused 3D feature point cloud. The trajectory reconstruction module is used to generate high-speed three-dimensional point trajectory data based on the fused three-dimensional feature point cloud through a time-correlation transformer.

7. A terminal, characterized in that, The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the high-speed motion three-dimensional point tracking and trajectory reconstruction method as described in any one of claims 1 to 5; the processors are used to execute the programs.

8. A computer-readable storage medium storing a plurality of instructions, characterized in that, The instructions are applicable to be loaded and executed by a processor to implement the steps of the high-speed motion three-dimensional point tracking and trajectory reconstruction method as described in any one of claims 1 to 5.

Images