Three-dimensional reconstruction method and device for projectile motion based on three-dimensional Gaussian splashing

Abstract

The invention relates to the technical field of three-dimensional reconstruction, in particular to a projectile motion three-dimensional reconstruction method and device based on three-dimensional Gaussian splashing, and the method comprises the steps: collecting and preprocessing a simulation data set containing an RGB image sequence and an asynchronous event stream, so as to generate an event representation diagram; jointly segmenting the event representation graph to obtain a background and a projectile target; establishing a 3D normalized coordinate system taking the projectile target as the center to obtain an initial geometric reference; based on a 3D Gaussian splashing method, generating a Gaussian kernel rendering image according to the initial geometric reference, and performing multi-modal joint loss optimization and Gaussian kernel screening on the Gaussian kernel rendering image to obtain a static Gaussian field; and mapping the static Gaussian field to a real coordinate system, and fusing multiple modes, gravity constraint and optical flow smooth loss to optimize the camera pose so as to generate projectile motion parameters. Therefore, the problems that in a traditional reconstruction method, the edge of a high-speed target is fuzzy, optimization is difficult due to no initial geometric reference, and the motion trail lacks physical consistency are solved.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and in particular to a method and apparatus for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing. Background Technology

[0002] 3D reconstruction of projectile motion has crucial practical applications in numerous industries and research fields. In the military, accurate measurement of ballistic trajectories is key to evaluating weapon performance and making trajectory corrections. In sports research, 3D modeling of projectile motion generated by athletes' throwing and striking (such as the flight paths of baseballs and javelins) provides quantifiable data for optimizing technical movements, helping athletes improve their competitive abilities. In industrial production, monitoring the motion parameters of materials during throwing is a vital step in maintaining stable production processes and improving efficiency, effectively preventing equipment failures or production accidents caused by abnormal material movement. In robotics, real-time 3D reconstruction and motion prediction of dynamic projectiles are fundamental for robots to achieve precise grasping and interception, directly impacting the safety and efficiency of human-robot collaboration. Developing high-precision, high-efficiency 3D reconstruction technology for projectile motion is of great significance for promoting technological development in related fields.

[0003] Image-frame-based projectile motion modeling technology acquires data using RGB cameras or multi-view vision systems, then extracts feature points (such as contours and textures) from continuously captured image frames. Motion parameters are estimated using feature matching between frames, and finally, 3D reconstruction is achieved by combining algorithms such as triangulation and bundle adjustment. However, this technology has significant drawbacks: high-speed projectiles experience significant displacement during camera exposure, causing motion blur and severely impacting the accuracy and stability of feature point extraction; traditional cameras acquire data at a fixed frame rate, resulting in a large amount of redundant information that increases storage and transmission costs and reduces subsequent processing efficiency; the 3D reconstruction process of multi-view systems requires complex global optimization, and the computational load increases with the number of viewpoints, making it difficult to meet real-time requirements and limiting its application in scenarios with high response speed requirements, such as dynamic robotic grasping. Summary of the Invention

[0004] This invention provides a method and apparatus for 3D reconstruction of projectile motion based on 3D Gaussian Splatting (3DGS), in order to solve problems such as blurred edges of high-speed targets, difficulty in optimization due to lack of initial geometric reference, and lack of physical consistency of motion trajectory in traditional reconstruction methods.

[0005] A first aspect of the present invention provides a method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing, comprising the following steps: A simulation dataset containing RGB image sequences and asynchronous event streams is collected and preprocessed to obtain an event representation map. The event representation map is then jointly segmented to obtain the background and the projectile target. A 3D normalized coordinate system centered on the projectile target is established. An initial geometric reference is obtained based on the 3D normalized coordinate system. Using a 3D Gaussian splashing method, a Gaussian kernel rendered image is generated based on the initial geometric reference, and multimodal joint loss optimization and Gaussian kernel filtering are performed on the Gaussian kernel rendered image to obtain a static Gaussian field. The static Gaussian field is mapped to the real coordinate system to obtain a reference Gaussian field. Based on the reference Gaussian field, the camera pose is optimized by fusing the multimodal loss, gravity constraint loss, and optical flow smoothing loss to generate projectile motion parameters.

[0006] Optionally, the acquisition and preprocessing of the simulation dataset containing RGB image sequences and asynchronous event streams to obtain an event representation graph includes: High-speed target motion features are extracted from the target projectile motion database to generate a simulation dataset containing RGB image sequences and asynchronous event streams. A synchronous acquisition system consisting of RGB cameras and event cameras is built. Under multiple scene, multiple lighting, and multiple motion modes, pre-calibrated camera parameters are called to perform spatial alignment with the RGB image sequences and asynchronous event streams in the simulation dataset. Then, the synchronous acquisition system is used to synchronously acquire RGB images and event stream data of real projectile motion, and the asynchronous event streams are accumulated into the time window corresponding to each frame of RGB image to generate the event representation map.

[0007] Optionally, establishing a 3D normalized coordinate system centered on the projectile target includes: Calculate the 2D centroid of the projectile in each frame of the event representation image, and translate the 2D centroids of all frames to the target of the projectile; take the translated 2D centroid as the origin, the x-axis is parallel to the horizontal direction of the image, the y-axis is parallel to the vertical direction of the image, and the z-axis is perpendicular to the image plane to form the 3D normalized coordinate system.

[0008] Optionally, obtaining the initial geometric reference according to the 3D normalized coordinate system includes: The event representation map is used to estimate the coarse extrinsic parameters of the camera in different frames, and a sparse 3D point cloud of the scene is generated. Based on the 3D normalized coordinate system, the x, y, and z axis coordinate extreme values ​​of the sparse 3D point cloud of the scene are used as the basis, and each direction is expanded by a preset percentage to form a 3D bounding box containing the complete geometry of the parabolic body. The 3D bounding box containing the complete geometry of the parabolic body is used as the initial geometric reference.

[0009] Optionally, the 3D Gaussian splashing method generates the Gaussian kernel rendered image based on the initial geometric reference, and performs multimodal joint loss optimization and Gaussian kernel selection on the Gaussian kernel rendered image to obtain a static Gaussian field, including: Multiple 3D Gaussian kernels are randomly generated in the initial geometric reference, and the initial parameters of each 3D Gaussian kernel are: mean value. Uniformly distributed within the 3D bounding box of the parabolic body, with the covariance matrix initially being a diagonal matrix. The colors are sampled from the parabolic region of the RGB image, and the transparency is uniformly set to 0.5. The multiple 3D Gaussian kernels are processed using the 3D Gaussian splashing method to generate the Gaussian kernel rendered image. The Gaussian kernel rendered image is then subjected to multimodal joint loss optimization and Gaussian kernel filtering to obtain the static Gaussian field. The multimodal joint loss includes RGB rendering loss, optical flow smoothing loss, and event dynamic loss.

[0010] Optionally, the step of optimizing the camera pose based on the reference Gaussian field by fusing multimodal loss, gravity constraint loss, and optical flow smoothing loss to generate projectile motion parameters includes: Using the reference Gaussian field as the initial geometric model, the camera pose corresponding to each frame in the RGB image sequence is learned frame by frame until the multimodal loss, the gravity constraint loss and the optical flow smoothing loss meet the preset requirements, and the projectile motion parameters are output.

[0011] A second aspect of the present invention provides a 3D reconstruction device for projectile motion based on 3D Gaussian splashing, comprising: The system comprises the following modules: an acquisition module for acquiring and preprocessing a simulation dataset containing RGB image sequences and asynchronous event streams to obtain an event representation map; a segmentation module for jointly segmenting the event representation map to obtain the background and the projectile target; a construction module for establishing a 3D normalized coordinate system centered on the projectile target; an acquisition module for obtaining an initial geometric reference based on the 3D normalized coordinate system; a generation module for generating a Gaussian kernel rendered image based on the initial geometric reference using a 3D Gaussian splashing method, and performing multimodal joint loss optimization and Gaussian kernel filtering on the Gaussian kernel rendered image to obtain a static Gaussian field; a mapping module for mapping the static Gaussian field to the real coordinate system to obtain a reference Gaussian field; and an optimization module for optimizing the camera pose based on the reference Gaussian field by fusing the multimodal loss, gravity constraint loss, and optical flow smoothing loss to generate projectile motion parameters.

[0012] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional reconstruction method for projectile motion based on three-dimensional Gaussian splashing as described in the above embodiments.

[0013] A fourth aspect of the present invention provides a computer program product, which, when executed by a processor, implements the above-described method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing.

[0014] A fifth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing.

[0015] The present invention proposes a method and apparatus for 3D reconstruction of projectile motion based on 3D Gaussian splashing. First, dual-modal data is simultaneously acquired using an RGB camera and an event camera. Spatial alignment and event accumulation generate an event representation map to compensate for the high-speed motion blur defect of RGB images. Next, a pre-trained segmentation model is used to segment the dual-modal data, separating the projectile target from the background, and a 3D normalized coordinate system is constructed with the 2D centroid of the projectile as the origin to eliminate translational interference. Then, an initial geometric reference is obtained through COLMAP, and a 3D Gaussian kernel is initialized within a 3D bounding box. The Gaussian field parameters are optimized using multimodal joint losses (RGB rendering loss, optical flow loss, and event dynamic loss), and affine transformation is used to align the static Gaussian field with the real scale. Finally, based on the reference Gaussian field, multimodal joint losses, gravity constraint losses, and optical flow smoothing losses are fused to optimize the camera pose frame by frame, outputting the 3D position, trajectory, velocity, and other motion parameters of the projectile. This method can achieve 3D reconstruction of projectile motion in high-speed, weakly textured, and complex lighting scenes. Reconstruction and dynamic tracking can be applied to fields such as ballistic analysis, robot dynamic grasping, and sports training monitoring. Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0016] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a three-dimensional reconstruction method for projectile motion based on three-dimensional Gaussian splashing, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the specific execution of a three-dimensional reconstruction method for projectile motion based on three-dimensional Gaussian splashing, provided in an embodiment of the present invention. Figure 3 A block diagram illustrating a 3D reconstruction device for projectile motion based on 3D Gaussian splashing, provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0017] Explanation of reference numerals in the attached figures: 30. 3D reconstruction device for projectile motion based on 3D Gaussian splashing, 301-acquisition module, 302-segmentation module, 303-construction module, 304-acquisition module, 305-generation module, 306-mapping module, 307-optimization module, 307, 401-memory, 402-processor and 403-communication interface. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0019] The following describes a method and apparatus for 3D reconstruction of projectile motion based on 3D Gaussian splashing, according to embodiments of the present invention, with reference to the accompanying drawings. Addressing the problems mentioned in the background section regarding the blurring of high-speed target edges, difficulty in optimization due to the lack of initial geometric reference, and lack of physical consistency in motion trajectories in traditional reconstruction methods, the present invention provides a method for 3D reconstruction of projectile motion based on 3D Gaussian splashing. This method combines an event camera and 3DGS, and through a complete workflow design including data acquisition, target segmentation, coordinate system normalization, 3D modeling, and dynamic tracking, achieves high-precision, physically consistent 3D geometric and motion parameter estimation of the projectile.

[0020] Among these methods, the use of event cameras for projectile motion modeling offers a new direction for addressing the problems of traditional techniques. Event cameras are a novel type of dynamic visual sensor inspired by biological vision systems, and their operation differs significantly from traditional cameras: instead of outputting complete images at a fixed frame rate, they output sparse data containing spatiotemporal coordinates and the polarity of brightness changes as "events" when pixels detect changes in brightness. This asynchronous sampling characteristic gives them microsecond-level temporal resolution, enabling them to accurately capture the instantaneous changes in high-speed projectile motion and fundamentally solve the problem of motion blur. Simultaneously, event cameras have an extremely high dynamic range, operating stably in complex lighting environments such as strong and weak light, reducing data loss due to exposure issues. Furthermore, their data output is far lower than that of traditional cameras, greatly reducing data redundancy and laying the foundation for efficient subsequent processing. These advantages give event cameras a unique technological edge in the field of high-speed projectile motion modeling.

[0021] 3D Gaussian Splashing is a real-time radiation field rendering technique that uses explicit 3D Gaussian distributions to represent scenes, replacing traditional polygonal meshes or implicit neural radiation fields. Its core idea originates from splashing techniques in computer graphics, projecting discrete 3D Gaussians onto a 2D image plane for rasterization rendering. Compared to traditional point clouds, mesh models, and neural radiation fields, 3DGS has three core advantages: First, high rendering efficiency, supporting high-resolution rendering at 30fps+ thanks to differentiable rasterization algorithms, significantly outperforming NeRF; second, flexible dynamic modeling, supporting continuous optimization of the position, scale, and rotation parameters of the Gaussian distribution over time, accurately representing the continuity of projectile motion and effectively avoiding trajectory distortion caused by traditional keyframe interpolation; third, physically interpretable parameters, with each Gaussian distribution carrying physical properties such as position coordinates changing over time, which can be directly correlated with projectile motion parameters (velocity, acceleration) without additional feature transformation steps.

[0022] Specifically, Figure 1 This is a flowchart illustrating a three-dimensional reconstruction method for projectile motion based on three-dimensional Gaussian splashing, provided in an embodiment of the present invention.

[0023] like Figure 1 As shown, the three-dimensional reconstruction method for projectile motion based on three-dimensional Gaussian splashing includes the following steps: In step S101, a simulation dataset containing RGB image sequences and asynchronous event streams is acquired and preprocessed to obtain an event representation graph.

[0024] In some embodiments, a simulation dataset containing RGB image sequences and asynchronous event streams is acquired and preprocessed to obtain an event representation graph, including: High-speed target motion features are extracted from the target projectile motion database to generate a simulation dataset containing RGB image sequences and asynchronous event streams; Build a synchronous acquisition system consisting of RGB cameras and event cameras; In multiple scene, multiple lighting, and multiple motion modes, the pre-calibrated camera parameters are called and spatially aligned with the RGB image sequence and asynchronous event stream in the simulation dataset. Then, the synchronous acquisition system is used to synchronously acquire the RGB images and event stream data of the real projectile motion, and the asynchronous event stream is accumulated into the time window corresponding to each frame of RGB image to generate an event representation map.

[0025] like Figure 2 As shown, in actual execution, this embodiment of the invention acquires dual-modal data covering multiple scenarios. On the one hand, it extracts high-speed target motion features from existing projectile motion databases to generate a simulation dataset containing RGB image sequences and asynchronous event streams, with each sample corresponding to the projectile motion process in the same scenario. On the other hand, it builds a synchronous acquisition hardware system composed of RGB cameras and event cameras to synchronously acquire RGB images and event stream data of real projectile motion under different scenarios (such as indoor and outdoor), different lighting conditions (such as strong light, backlight, and normal lighting), and different motion modes (such as horizontal projectile, oblique projectile, and rotating projectile). During the acquisition process, the intrinsic parameters of the camera (such as the focal length and principal point coordinates of the RGB camera, and the pixel scale of the event camera) and the extrinsic parameters (rotation matrix and translation vector between the two cameras) obtained in advance through calibration tools are first called to perform spatial coordinate calibration on the event data and RGB images. This ensures that the spatial position of the same projectile target is completely matched in the two modal data, avoiding target misalignment caused by equipment installation pose deviation. Then, based on the synchronous trigger signal output by the hardware system, the asynchronous event stream output by the event camera is accumulated according to the time window corresponding to each frame of the RGB image (the duration of the time window is the reciprocal of the RGB camera frame rate). The sparse event stream is transformed into a frame-level event representation map that corresponds one-to-one with the RGB frames, forming a pair of "RGB image - event representation map" dual-modal data.

[0026] In step S102, the event representation map is jointly segmented to obtain the background and the projectile target.

[0027] like Figure 2As shown, in the actual execution process, a pre-trained image segmentation model is used. Preprocessed bimodal data (RGB image frames and corresponding event representation maps) are used as model input to separate the projectile target from the background. Specifically, since the event representation map can accurately capture the dynamic edge features of high-speed targets, even if the RGB image shows blurred edges due to the high-speed motion of the projectile, the event representation map can still clearly present the target outline. Therefore, by jointly inputting multimodal data, the segmentation model can more accurately identify the projectile region, effectively reducing the interference of background pixels on the segmentation result, and outputting only a binary mask containing the projectile target. Based on the segmented projectile mask, the projectile region in the RGB image and event representation map is cropped to remove irrelevant background areas, further focusing on the projectile target and reducing the computational load of subsequent 3D modeling.

[0028] In step S103, a 3D normalized coordinate system is established with the projectile target as the center.

[0029] In some embodiments, establishing a 3D normalized coordinate system centered on the projectile target includes: Calculate the 2D centroid of the projectile in each frame of the event representation graph, and translate the 2D centroid of the projectile in all frames to the projectile target. With the translated 2D centroid as the origin, the x-axis is parallel to the horizontal direction of the image, the y-axis is parallel to the vertical direction of the image, and the z-axis is perpendicular to the image plane, forming a 3D normalized coordinate system.

[0030] like Figure 2 As shown, in the actual execution process, in order to eliminate the interference of the projectile translational motion on 3D modeling, a 3D normalized coordinate system centered on the projectile target is constructed: First, based on the segmented projectile mask, the mean value of all pixel coordinates within the projectile region of each frame is calculated, and the 2D centroid of the projectile in each frame of the event representation map is calculated. The 2D centroids of all projectiles in all frames are translated to the center of the image (i.e., the target of the projectile). Through image translation, the 2D centroids of all projectiles in all frames are uniformly moved to the center of the image, thus normalizing the position of the projectiles on the 2D plane.

[0031] Based on this, a 3D coordinate system is defined: with the translated 2D centroid as the origin, the x-axis parallel to the horizontal direction of the image, the y-axis parallel to the vertical direction of the image, and the z-axis perpendicular to the image plane (pointing towards the camera), forming a 3D normalized coordinate system, thereby eliminating the translational displacement of the target in the feature space. The cropped bimodal data and the subsequently generated 3D model parameters are both mapped to this coordinate system, achieving motion decoupling of the projectile motion and providing a unified spatial reference for 3D modeling.

[0032] In step S104, the initial geometric datum is obtained according to the 3D normalized coordinate system.

[0033] In some embodiments, obtaining an initial geometric reference based on a 3D normalized coordinate system includes: The event representation graph is used to estimate the coarse extrinsic parameters of the camera in different frames, while generating a sparse 3D point cloud of the scene. Based on the 3D normalized coordinate system, the x, y, and z axis coordinate extreme values ​​of the sparse 3D point cloud of the scene are used as the basis, and each direction is expanded by a preset percentage to form a 3D bounding box containing the complete geometric shape of the parabolic body. The 3D bounding box containing the complete geometric shape of the parabolic body is used as the initial geometric reference.

[0034] like Figure 2 As shown, in actual execution, before optimizing the 3D Gaussian field parameters, it is necessary to obtain the initial geometric reference through the Structure for Motion (SFM) method to provide reasonable spatial constraints for Gaussian field initialization and avoid optimization convergence difficulties caused by initial parameter randomization. Specifically, the preprocessed RGB image sequence in the event representation map is selected and input into the COLMAP open-source tool. Through the tool's built-in feature extraction and matching algorithms (such as SIFT feature matching), the coarse extrinsic parameters of the camera in different frames (including the rotation matrix) are estimated. Translation vector At the same time, it generates a sparse 3D point cloud of the scene.

[0035] Furthermore, based on the 3D normalized coordinate system, and according to the spatial distribution range of the points in the sparse 3D point cloud, the initial boundary of the 3D bounding box of the projectile is determined: first, the extreme values ​​(i.e., maximum and minimum values) of the coordinates of all projectile points in the point cloud on the x, y, and z axes are counted, and then the range of each coordinate axis direction is expanded to both sides by 15% to form a 3D bounding box containing the complete geometric shape of the projectile as the initial geometric reference.

[0036] In step S105, a Gaussian kernel rendering image is generated based on the 3D Gaussian splashing method according to the initial geometric reference, and the Gaussian kernel rendering image is subjected to multimodal joint loss optimization and Gaussian kernel screening to obtain a static Gaussian field.

[0037] In some embodiments, based on the 3D Gaussian splashing method, a Gaussian kernel rendering image is generated according to an initial geometric benchmark, and multimodal joint loss optimization and Gaussian kernel selection are performed on the Gaussian kernel rendering image to obtain a static Gaussian field, including: Multiple 3D Gaussian kernels are randomly generated from the initial geometric datum, and the initial parameters of each 3D Gaussian kernel are: mean value Uniformly distributed within the 3D bounding box of the parabolic body, with the covariance matrix initially being a diagonal matrix. Colors are sampled from the parabolic region of the RGB image, and the transparency is uniformly set to 0.5. Multiple 3D Gaussian kernels are processed using the 3D Gaussian splashing method to generate Gaussian kernel rendered images; Multimodal joint loss optimization and Gaussian kernel selection are performed on the Gaussian kernel rendered image to obtain a static Gaussian field. The multimodal joint loss includes RGB rendering loss, optical flow smoothing loss and event dynamic loss.

[0038] like Figure 2 As shown, in the actual execution process, during the Gaussian field initialization phase, according to the requirements of the 3D Gaussian splashing method, N 3D Gaussian kernels are randomly generated in the initial geometric reference. The initial parameters of each kernel are: mean value... Uniformly distributed within the 3D bounding box of the parabolic body (based on the 2D clipping region extending the z-axis range) to ensure that the Gaussian kernel energy covers the complete geometric range of the parabolic body in the initial stage. Covariance matrix. Initially set as a diagonal matrix This ensures isotropy in the initial state. The color of the Gaussian kernel. Randomly sample from the parabolic region of the RGB image to ensure the initial color matches the true appearance of the parabolic body. Transparency. The value is uniformly set to 0.5 to ensure that the Gaussian kernel has basic visibility in the initial stage.

[0039] In the parameter optimization stage, multiple 3D Gaussian kernels are processed using the 3D Gaussian splashing method to generate a Gaussian kernel rendered image. The Gaussian kernel rendered image is then optimized using a total optimization loss function that integrates multimodal joint loss function, RGB texture constraints, optical flow smoothing constraints, and event dynamic constraints. During optimization, Gaussian kernels with excessively large maximum eigenvalues ​​in the covariance matrix are pruned using axial constraints (eliminating boundary jaggedness), and isolated Gaussian kernels are removed using an outlier removal algorithm (preventing Gaussian kernels from being scattered in the background area), ensuring that the Gaussian kernels are concentrated on the projectile body, ultimately yielding a static Gaussian field. The specific expression for the overall optimization loss function is as follows:

[0040] In the formula, As an RGB texture constraint loss, it renders a 2D image using a 3D Gaussian field and calculates the L1 loss between the rendered image and the real RGB image. This loss ensures that the appearance of the 3D Gaussian field rendered image is consistent with the real texture of the parabolic body. A true RGB image. Render the image using a Gaussian field; To mitigate the optical flow smoothing loss, the E-RAFT algorithm is used to calculate the optical flow between adjacent frames from the RGB image and the event stream. Then, the RAFT algorithm is used to calculate the predicted optical flow between corresponding images rendered with a 3D Gaussian kernel. To calculate the loss Constrain the motion continuity of the Gaussian kernel to avoid motion breaks between frames; The event dynamic constraint loss is used to strengthen dynamic edge constraints. It calculates the brightness changes between adjacent frames based on the 3D Gaussian kernel-rendered image. By comparing the polarity of brightness changes recorded in the event, the loss is calculated using the loss function. To enhance the modeling accuracy of the dynamic edges of a projectile, thereby improving the geometric accuracy of the projectile's edges, among other things, For a single event in the event stream, The polarity of the event; , , These are the weighting coefficients.

[0041] In step S106, the static Gaussian field is mapped to the real coordinate system to obtain the reference Gaussian field.

[0042] like Figure 2 As shown, in actual implementation, an affine transformation involving rotation, translation, and scaling is learned. (r is the quaternion rotation, t is the translation vector, and s is the scale factor), to transform the static Gaussian field The (normalized scale) transformation is applied to the coordinate system corresponding to the original image frame (first frame) to obtain the aligned Gaussian field. ; the aligned Gaussian field The image is rendered and compared with the original RGB first frame image and the corresponding event stream. The multimodal joint loss function in step S105 is reused to calculate the difference between the rendered result and the real data. The affine transformation parameters are iteratively optimized and adjusted until the loss between the rendered result and the real data converges, resulting in a reference Gaussian field that matches the real scene after transformation. This can provide an initial geometric reference Gaussian field for subsequent dynamic tracking.

[0043] In step S107, the camera pose is optimized by fusing multimodal loss, gravity constraint loss and optical flow smoothing loss based on the reference Gaussian field to generate projectile motion parameters.

[0044] In some embodiments, based on a reference Gaussian field, the camera pose is optimized by fusing multimodal loss, gravity constraint loss, and optical flow smoothing loss to generate projectile motion parameters, including: Using a reference Gaussian field as the initial geometric model, the camera pose corresponding to each frame in the RGB image sequence is learned frame by frame until the multimodal loss, gravity constraint loss and optical flow smoothing loss meet the preset requirements, and the projectile motion parameters are output.

[0045] like Figure 2 As shown, in actual implementation, based on the reference Gaussian field The motion parameters (i.e., camera pose) of each frame of the image are learned frame by frame: For each frame of the image to be processed, the reference Gaussian field is used as the initial geometric model. The spatial pose of the Gaussian field is iteratively adjusted by optimizing the SE-3 transformation (including rotation and translation). Based on the preset total loss function, the matching degree between the adjusted Gaussian field rendering image and the current frame RGB image and event stream is maximized. Finally, the 3D position, motion trajectory, velocity, acceleration and other physical parameters of each frame of the projectile are output.

[0046] In the optimization process, in addition to continuing to use the multimodal joint loss function in step S105, physical constraints (gravitational acceleration constraints) and optical flow smoothing constraints are introduced as the total loss function for dynamic tracking to improve the accuracy and stability of motion parameters.

[0047] The gravitational acceleration constraint is introduced based on Newtonian mechanics principles to ensure the physical consistency of projectile motion. Specifically, it is first determined based on the radius of the 3D Gaussian kernel. Spatial location Through the weighted average formula Calculate the 3D center of mass of a projectile The centroid acceleration is calculated by the second-order difference of the centroid displacement between frames. ( (This is the inter-frame time interval). Let the unit vector of gravity direction in the camera coordinate system be... ( (where gravitational acceleration is constant), and acceleration Decomposed into parallel The amount and vertical components Then the gravity constraint loss is calculated. ,in An adaptive regularization term is used to avoid loss explosion when acceleration approaches zero; The rate of change of the vertical component is used to force smoothing of the vertical acceleration (without additional external force). ).

[0048] The optical flow smoothing constraint is as follows: To avoid inter-frame jitter in motion tracking, the difference between the optical flow of the 3D Gaussian field rendered image and the real optical flow (obtained by the E-RAFT algorithm) is calculated, and a smoothing penalty is applied to the optical flow field gradient. The loss function is:

[0049] in, For optical flow gradient, To render image gradients, the optical flow smoothing weights in the image edge regions are adjusted using an exponential term to prevent edge details from being distorted due to optical flow smoothing. By fusing gravity loss, optical flow smoothing loss, and the loss from the previous step S105, the final output is physically consistent parameters such as the 3D position and motion trajectory of the projectile in each frame.

[0050] In summary, the three-dimensional reconstruction method for projectile motion based on three-dimensional Gaussian splashing proposed in the embodiments of the present invention has the following beneficial effects: (1) By combining projectile images captured by a regular camera and event stream data captured by an event camera, the present invention can more accurately locate high-speed projectile targets, especially in scenarios where the rapid movement of the projectile causes the edges of the regular image to become blurred and details to be lost. It can effectively compensate for image defects and clearly separate the projectile from the background. (2) By constructing a 3D Gaussian model from the projectile image and the corresponding event stream, and by adjusting the model parameters in combination with RGB appearance, motion smoothness, dynamic edge and other constraints, the model can accurately reconstruct the 3D form of the projectile by combining the temporal information of any frame image. The introduction of the 3D Gaussian model makes the model fit the real geometric structure of the projectile better, and enhances the ability to restore the outline and details of the projectile, especially the projectile with weak texture and high speed. (3) By introducing gravity physical constraints and optical flow smoothing optimization, and combining the physical laws of projectile motion to finely adjust the preliminary tracking results, the trajectory deviation can be corrected for the motion state of the projectile in each frame, avoiding screen jitter during the tracking process and improving the accuracy of projectile motion parameters (position, velocity) estimation.

[0051] Next, with reference to the accompanying drawings, a 3D reconstruction device for projectile motion based on 3D Gaussian splashing according to an embodiment of the present invention is described.

[0052] Figure 3 This is a block diagram of a 3D reconstruction device for projectile motion based on 3D Gaussian splashing, provided in an embodiment of the present invention.

[0053] like Figure 3 As shown, the 3D reconstruction device 30 based on 3D Gaussian splashing projectile motion includes: acquisition module 301, segmentation module 302, construction module 303, acquisition module 304, generation module 305, mapping module 306 and optimization module 307.

[0054] The system comprises the following modules: Acquisition module 301 acquires and preprocesses a simulation dataset containing RGB image sequences and asynchronous event streams to obtain an event representation map; Segmentation module 302 performs joint segmentation on the event representation map to obtain the background and projectile target; Construction module 303 establishes a 3D normalized coordinate system centered on the projectile target; Acquisition module 304 acquires the initial geometric reference based on the 3D normalized coordinate system; Generation module 305 generates a Gaussian kernel rendered image based on the initial geometric reference using a 3D Gaussian splashing method, and performs multimodal joint loss optimization and Gaussian kernel filtering on the Gaussian kernel rendered image to obtain a static Gaussian field; Mapping module 306 maps the static Gaussian field to the real coordinate system to obtain a reference Gaussian field; and Optimization module 307 optimizes the camera pose based on the reference Gaussian field by fusing multimodal loss, gravity constraint loss, and optical flow smoothing loss to generate projectile motion parameters.

[0055] In some embodiments, the acquisition module 301 includes: The extraction unit is used to extract high-speed target motion features from the target projectile motion database to generate a simulation dataset containing RGB image sequences and asynchronous event streams; The building unit is used to build a synchronous acquisition system consisting of RGB cameras and event cameras; The generation unit is used to spatially align the pre-calibrated camera parameters with the RGB image sequence and asynchronous event stream in the simulation dataset under multiple scene, multiple lighting, and multiple motion modes. Then, it uses a synchronous acquisition system to synchronously acquire RGB images and event stream data of the real projectile motion and accumulates the asynchronous event stream into the time window corresponding to each frame of RGB image to generate an event representation map.

[0056] In some embodiments, the construction module 303 includes: Translation unit is used to calculate the 2D centroid of the projectile in each frame of the event representation graph and translate the 2D centroid of the projectile in all frames to the projectile target. The building unit is used to form a 3D normalized coordinate system with the translated 2D centroid as the origin, the x-axis parallel to the horizontal direction of the image, the y-axis parallel to the vertical direction of the image, and the z-axis perpendicular to the image plane.

[0057] In some embodiments, the acquisition module 304 includes: The estimation unit is used to estimate the coarse extrinsic parameters of the camera in different frames using the event representation map, while generating a sparse 3D point cloud of the scene. The forming unit is used to expand by a preset percentage in each direction based on the extreme values ​​of the x, y, and z axis coordinates of the sparse 3D point cloud of the scene, using a 3D normalized coordinate system, to form a 3D bounding box containing the complete geometry of the parabolic body, and to use the 3D bounding box containing the complete geometry of the parabolic body as the initial geometric reference.

[0058] In some embodiments, the generation module 305 includes: An initialization unit is used to randomly generate multiple 3D Gaussian kernels from the initial geometric datum, and the initial parameters of each 3D Gaussian kernel are: mean value... Uniformly distributed within the 3D bounding box of the parabolic body, with the covariance matrix initially being a diagonal matrix. Colors are sampled from the parabolic region of the RGB image, and the transparency is uniformly set to 0.5. The generation unit is used to process multiple 3D Gaussian kernels using the 3D Gaussian splashing method to generate a Gaussian kernel rendered image. The optimization and filtering unit is used to perform multimodal joint loss optimization and Gaussian kernel filtering on the Gaussian kernel rendered image to obtain a static Gaussian field. The multimodal joint loss includes RGB rendering loss, optical flow smoothing loss and event dynamic loss.

[0059] In some embodiments, the optimization module 307 includes: Using a reference Gaussian field as the initial geometric model, the camera pose corresponding to each frame in the RGB image sequence is learned frame by frame until the multimodal loss, gravity constraint loss and optical flow smoothing loss meet the preset requirements, and the projectile motion parameters are output.

[0060] It should be noted that the foregoing explanation of the embodiment of the projectile motion 3D reconstruction method based on 3D Gaussian splashing also applies to the projectile motion 3D reconstruction device based on 3D Gaussian splashing in this embodiment, and will not be repeated here.

[0061] The 3D reconstruction device for projectile motion based on 3D Gaussian splashing proposed in the embodiments of the present invention has the following beneficial effects: (1) By combining projectile images captured by a regular camera and event stream data captured by an event camera, the present invention can more accurately locate high-speed projectile targets, especially in scenarios where the rapid movement of the projectile causes the edges of the regular image to become blurred and details to be lost. It can effectively compensate for image defects and clearly separate the projectile from the background. (2) By constructing a 3D Gaussian model from the projectile image and the corresponding event stream, and by adjusting the model parameters in combination with RGB appearance, motion smoothness, dynamic edge and other constraints, the model can accurately reconstruct the 3D form of the projectile by combining the temporal information of any frame image. The introduction of the 3D Gaussian model makes the model fit the real geometric structure of the projectile better, and enhances the ability to restore the outline and details of the projectile, especially the projectile with weak texture and high speed. (3) By introducing gravity physical constraints and optical flow smoothing optimization, and combining the physical laws of projectile motion to finely adjust the preliminary tracking results, the trajectory deviation can be corrected for the motion state of the projectile in each frame, avoiding screen jitter during the tracking process and improving the accuracy of projectile motion parameters (position, velocity) estimation.

[0062] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include: The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0063] When the processor 402 executes the program, it implements the three-dimensional reconstruction method of projectile motion based on three-dimensional Gaussian splashing provided in the above embodiments.

[0064] Furthermore, electronic devices also include: Communication interface 403 is used for communication between memory 401 and processor 402.

[0065] The memory 401 is used to store computer programs that can run on the processor 402.

[0066] Memory 401 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0067] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0068] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0069] Processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0070] This invention also provides a computer program product, which, when executed by a processor, implements the above-described method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing.

[0071] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing.

[0072] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0073] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0074] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0075] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0076] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0077] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0078] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0079] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing, characterized in that, Includes the following steps: A simulation dataset containing RGB image sequences and asynchronous event streams was collected and preprocessed to obtain an event representation graph; The event representation map is jointly segmented to obtain the background and the projectile target; Establish a 3D normalized coordinate system centered on the projectile target; The initial geometric reference is obtained based on the 3D normalized coordinate system; Based on the 3D Gaussian splashing method, the Gaussian kernel rendering image is generated according to the initial geometric benchmark, and the Gaussian kernel rendering image is subjected to multimodal joint loss optimization and Gaussian kernel selection to obtain a static Gaussian field. The static Gaussian field is mapped to the real coordinate system to obtain the reference Gaussian field; Based on the reference Gaussian field, the camera pose is optimized by fusing the multimodal loss, gravity constraint loss, and optical flow smoothing loss to generate projectile motion parameters.

2. The method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The acquisition and preprocessing of the simulation dataset containing RGB image sequences and asynchronous event streams to obtain an event representation graph includes: High-speed target motion features are extracted from a target projectile motion database to generate a simulation dataset containing RGB image sequences and asynchronous event streams; Build a synchronous acquisition system consisting of RGB cameras and event cameras; In multiple scene, multiple lighting, and multiple motion modes, the pre-calibrated camera parameters are called and spatially aligned with the RGB image sequence and asynchronous event stream in the simulation dataset. Then, the synchronous acquisition system is used to synchronously acquire the RGB images and event stream data of the real projectile motion, and the asynchronous event stream is accumulated into the time window corresponding to each frame of RGB image to generate the event representation map.

3. The method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The establishment of a 3D normalized coordinate system centered on the projectile target includes: Calculate the 2D centroid of the projectile in each frame of the event representation graph, and translate the 2D centroid of the projectile in all frames to the projectile target. With the translated 2D centroid as the origin, the x-axis is parallel to the horizontal direction of the image, the y-axis is parallel to the vertical direction of the image, and the z-axis is perpendicular to the image plane, forming the 3D normalized coordinate system.

4. The method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The step of obtaining the initial geometric reference according to the 3D normalized coordinate system includes: The event representation map is used to estimate the coarse extrinsic parameters of the camera in different frames, while generating a sparse 3D point cloud of the scene. Based on the 3D normalized coordinate system, and taking the extreme values ​​of the x, y, and z axis coordinates of the sparse 3D point cloud of the scene as a basis, each direction is expanded by a preset percentage to form a 3D bounding box containing the complete geometric shape of the parabolic body, and the 3D bounding box containing the complete geometric shape of the parabolic body is used as the initial geometric reference.

5. The method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The 3D Gaussian splashing method generates the Gaussian kernel rendered image based on the initial geometric reference, and performs multimodal joint loss optimization and Gaussian kernel selection on the Gaussian kernel rendered image to obtain a static Gaussian field, including: Multiple 3D Gaussian kernels are randomly generated in the initial geometric reference, and the initial parameters of each 3D Gaussian kernel are: mean value. Uniformly distributed within the 3D bounding box of the parabolic body, with the covariance matrix initially being a diagonal matrix. Colors are sampled from the parabolic region of the RGB image, and the transparency is uniformly set to 0.

5. The multiple 3D Gaussian kernels are processed using the 3D Gaussian splashing method to generate the Gaussian kernel rendered image; The Gaussian kernel rendered image is subjected to multimodal joint loss optimization and Gaussian kernel selection to obtain the static Gaussian field, wherein the multimodal joint loss includes RGB rendering loss, optical flow smoothing loss and event dynamic loss.

6. The method for three-dimensional reconstruction of projectile motion based on three-dimensional Gaussian splashing according to claim 5, characterized in that, The optimization of camera pose based on the reference Gaussian field, incorporating multimodal loss, gravity constraint loss, and optical flow smoothing loss, to generate projectile motion parameters includes: Using the reference Gaussian field as the initial geometric model, the camera pose corresponding to each frame in the RGB image sequence is learned frame by frame until the multimodal loss, the gravity constraint loss and the optical flow smoothing loss meet the preset requirements, and the projectile motion parameters are output.

7. A 3D reconstruction device for projectile motion based on 3D Gaussian splashing, characterized in that, include: The acquisition module is used to acquire and preprocess the simulation dataset containing RGB image sequences and asynchronous event streams to obtain the event representation map; A segmentation module is used to perform joint segmentation on the event representation map to obtain the background and the projectile target; The construction module is used to establish a 3D normalized coordinate system centered on the projectile target; The acquisition module is used to acquire the initial geometric reference according to the 3D normalized coordinate system; The generation module is used to generate the Gaussian kernel rendering image based on the initial geometric reference according to the 3D Gaussian splashing method, and to perform multimodal joint loss optimization and Gaussian kernel selection on the Gaussian kernel rendering image to obtain a static Gaussian field. A mapping module is used to map the static Gaussian field to the real coordinate system to obtain a reference Gaussian field; The optimization module is used to optimize the camera pose based on the reference Gaussian field, by fusing the multimodal loss, gravity constraint loss and optical flow smoothing loss, in order to generate projectile motion parameters.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional reconstruction method for projectile motion based on three-dimensional Gaussian splashing as described in any one of claims 1-6.

9. A computer program product, characterized in that, When the computer program / instruction is executed by the processor, it implements the three-dimensional reconstruction method for projectile motion based on three-dimensional Gaussian splashing as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the three-dimensional reconstruction method for projectile motion based on three-dimensional Gaussian splashing as described in any one of claims 1-6.

Images