Three-dimensional gaussian splatting method, system, electronic device and program product based on cross-modal trajectory constraints

Abstract

The disclosure provides a three-dimensional Gaussian splatting method, system, electronic device and program product based on cross-modal trajectory constraints. The three-dimensional Gaussian splatting method comprises: obtaining a first scale-free visual trajectory, a first scale-free sparse point cloud and a second scale-free metric trajectory corresponding to each other in a target scene; calculating target space transformation parameters between the first visual trajectory and the metric trajectory; based on the target space transformation parameters, processing the first visual trajectory and the first sparse point cloud to obtain a second visual trajectory with a scale and a second sparse point cloud; performing three-dimensional Gaussian splatting processing on the second visual trajectory and the second sparse point cloud to obtain a three-dimensional Gaussian splatting model with a scale. In this way, the three-dimensional Gaussian splatting model of the target scene can have a scale, and the direct three-dimensional Gaussian splatting model can be used in scenes requiring real physical scale, such as engineering measurement, geometric deviation analysis, construction progress monitoring, etc.

Description

Technical Field

[0001] This disclosure relates to the field of 3D scene reconstruction technology, and in particular to a 3D Gaussian splashing method, system, electronic device and program product based on cross-modal trajectory constraints. Background Technology

[0002] 3D Gaussian splashing technology is a novel perspective synthesis and reconstruction method that uses Gaussian primitives to represent 3D scenes, and it has high-fidelity rendering and real-time drawing capabilities.

[0003] However, current 3D Gaussian splashing methods can only initialize camera pose and sparse point clouds based on purely visual images. The reconstructed 3D Gaussian splashing model thus only retains the correct shape proportions but lacks true physical scale. This makes the reconstructed 3D Gaussian splashing model unsuitable for direct use in scenarios requiring true physical scale, such as engineering surveying, geometric deviation analysis, and construction progress monitoring. Therefore, 3D Gaussian splashing methods still need improvement. Summary of the Invention

[0004] The technical problem this disclosure aims to solve is to overcome the limitation that existing 3D Gaussian splashing methods only retain the correct shape and proportions but lack the true physical scale of the reconstructed 3D Gaussian splashing model. This makes the reconstructed 3D Gaussian splashing model unsuitable for direct use in scenarios requiring true physical scale, such as engineering surveying, geometric deviation analysis, and construction progress monitoring. This disclosure provides a 3D Gaussian splashing method, system, electronic device, and program product based on cross-modal trajectory constraints.

[0005] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0006] In a first aspect, this disclosure provides a three-dimensional Gaussian splashing method based on cross-modal trajectory constraints, the three-dimensional Gaussian splashing method comprising:

[0007] The system acquires a scale-free first visual trajectory, a scale-free first sparse point cloud, and a scaled metric trajectory that correspond to each other in the target scene. The first visual trajectory is used to characterize the motion path of the image acquisition device in the target scene, and the metric trajectory is used to characterize the motion path of the scale acquisition device in the target scene.

[0008] Calculate the target space transformation parameters between the first visual trajectory and the measurement trajectory; the target space transformation parameters are used to align the first visual trajectory and the measurement trajectory in spatial dimensions;

[0009] Based on the target space transformation parameters, the first visual trajectory and the first sparse point cloud are processed to obtain a second visual trajectory and a second sparse point cloud with scale.

[0010] The second visual trajectory and the second sparse point cloud are subjected to three-dimensional Gaussian splashing to obtain a three-dimensional Gaussian splashing model with scale.

[0011] Optionally, the step of obtaining the scale-free first visual trajectory, the scale-free first sparse point cloud, and the scaled metric trajectory corresponding to each other in the target scene includes:

[0012] Acquire a sequence of scene images of the target scene;

[0013] The scene image sequence is processed based on a sparse reconstruction algorithm to obtain the first visual trajectory and the first sparse point cloud.

[0014] Optionally, the step of calculating the target space transformation parameters between the first visual trajectory and the metric trajectory includes, prior to:

[0015] Obtain the timestamps of the first visual trajectory and the measurement trajectory respectively;

[0016] Based on the timestamp, the first visual trajectory and the metric trajectory are aligned in the time dimension.

[0017] Optionally, the step of calculating the target space transformation parameters between the first visual trajectory and the metric trajectory includes:

[0018] Based on a preset duration and a preset step size, the aligned first visual trajectory and the measurement trajectory are divided into several time windows;

[0019] In response to any time window, if the consistency deviation between the local relative motion of the first visual trajectory and the local relative motion of the measurement trajectory is greater than or equal to the deviation threshold, the first visual trajectory and the measurement trajectory within the corresponding time window are removed to obtain the first visual trajectory and the measurement trajectory after removal processing; wherein, the local relative motion represents the pose change amount jointly determined by multiple pose points of the trajectory and its corresponding trajectory within the time window.

[0020] The target space transformation parameters are calculated based on the first visual trajectory after the removal process and the metric trajectory after the removal process.

[0021] Optionally, the first visual trajectory after the removal process includes several first pose points, and the metric trajectory after the removal process includes several second pose points; and the first pose points and the second pose points correspond to each other.

[0022] The step of calculating the target space transformation parameters between the first visual trajectory and the metric trajectory includes:

[0023] Based on the first pose point and the second pose point, calculate the initial spatial transformation parameters between the first visual trajectory after culling and the metric trajectory after culling.

[0024] The target space transformation parameters are calculated based on the initial space transformation parameters.

[0025] Optionally, the step of calculating the target spatial transformation parameters based on the initial spatial transformation parameters includes:

[0026] Based on the initial spatial transformation parameters, a target loss function is constructed between the first visual trajectory and the metric trajectory;

[0027] The initial spatial transformation parameters are iteratively updated based on a preset iterative logic.

[0028] Based on the target loss function, calculate the loss value corresponding to the initial spatial transformation parameters after each iteration update;

[0029] In response to the loss value being less than or equal to a preset loss value, the initial spatial transformation parameter corresponding to the current loss value is used as the target spatial transformation parameter.

[0030] Optionally, the initial spatial transformation parameters include initial scale parameters, initial rotation parameters, and initial translation parameters;

[0031] The target loss function is:

[0032] ;

[0033] Wherein, ρ(·) is the robust loss between the first visual trajectory and the metric trajectory; The first visual trajectory; For the metric trajectory; The initial scale parameter; The initial rotation parameters are... The initial translation parameter is denoted as ; N is the total number of pose pairs formed by the first pose point and the second pose point; i∈[1,N].

[0034] Secondly, this disclosure also provides a three-dimensional Gaussian splashing system based on cross-modal trajectory constraints, the three-dimensional Gaussian splashing system comprising:

[0035] The acquisition module is used to acquire a scale-free first visual trajectory, a scale-free first sparse point cloud, and a scaled metric trajectory that correspond to each other in the target scene; wherein, the first visual trajectory is used to characterize the motion path of the image acquisition device in the target scene; and the metric trajectory is used to characterize the motion path of the scale acquisition device in the target scene.

[0036] The calculation module is used to calculate the target space transformation parameters between the first visual trajectory and the measurement trajectory; the target space transformation parameters are used to align the first visual trajectory and the measurement trajectory in spatial dimensions.

[0037] The processing module is used to process the first visual trajectory and the first sparse point cloud based on the target space transformation parameters to obtain a second visual trajectory and a second sparse point cloud with scale.

[0038] The processing module is also used to perform three-dimensional Gaussian splashing processing on the second visual trajectory and the second sparse point cloud to obtain a three-dimensional Gaussian splashing model with scale.

[0039] Optionally, the acquisition module includes:

[0040] The acquisition unit is used to acquire a sequence of scene images of the target scene;

[0041] The processing unit is used to process the scene image sequence based on a sparse reconstruction algorithm to obtain the first visual trajectory and the first sparse point cloud.

[0042] Optionally, the acquisition module is further configured to acquire the timestamps of the first visual trajectory and the measurement trajectory, respectively;

[0043] The three-dimensional Gaussian splashing system also includes:

[0044] The alignment module aligns the first visual trajectory and the metric trajectory in the time dimension based on the timestamp.

[0045] Optionally, the computing module includes:

[0046] The segmentation unit is used to divide the aligned first visual trajectory and the measurement trajectory into several time windows based on a preset duration and a preset step size;

[0047] The response unit, in response to any time window where the consistency deviation between the local relative motion of the first visual trajectory and the local relative motion of the measurement trajectory is greater than or equal to a deviation threshold, removes the first visual trajectory and the measurement trajectory within the corresponding time window to obtain the first visual trajectory and the measurement trajectory after removal processing; wherein, the local relative motion represents the pose change amount jointly determined by multiple pose points of the trajectory and its corresponding trajectory within the time window.

[0048] The calculation module is also used to calculate the target space transformation parameters based on the first visual trajectory after culling and the metric trajectory after culling.

[0049] Optionally, the first visual trajectory after the removal process includes several first pose points, and the metric trajectory after the removal process includes several second pose points; and the first pose points and the second pose points correspond to each other.

[0050] The calculation module also calculates the initial spatial transformation parameters between the first visual trajectory after culling and the metric trajectory after culling, based on the first pose point and the second pose point.

[0051] The calculation module also calculates the target space transformation parameters based on the initial space transformation parameters.

[0052] Optionally, the computing module further includes:

[0053] The construction unit is used to construct a target loss function between the first visual trajectory and the metric trajectory based on the initial spatial transformation parameters;

[0054] The update unit is used to iteratively update the initial spatial transformation parameters based on a preset iterative logic;

[0055] The calculation module also calculates the loss value corresponding to the initial spatial transformation parameters after each iteration update based on the target loss function;

[0056] The response unit also responds to the loss value being less than or equal to a preset loss value by using the initial spatial transformation parameter corresponding to the current loss value as the target spatial transformation parameter.

[0057] Optionally, the initial spatial transformation parameters include initial scale parameters, initial rotation parameters, and initial translation parameters;

[0058] The target loss function is:

[0059] ;

[0060] Wherein, ρ(·) is the robust loss between the first visual trajectory and the metric trajectory; The first visual trajectory; For the metric trajectory; The initial scale parameter; The initial rotation parameters are... The initial translation parameter is denoted as ; N is the total number of pose pairs formed by the first pose point and the second pose point; i∈[1,N].

[0061] Thirdly, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and used to run on the processor, wherein the processor executes the computer program to implement the three-dimensional Gaussian splashing method based on cross-modal trajectory constraints described above.

[0062] Fourthly, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned three-dimensional Gaussian splashing method based on cross-modal trajectory constraints.

[0063] Fifthly, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned three-dimensional Gaussian splashing method based on cross-modal trajectory constraints.

[0064] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0065] The significant advancements of this disclosure lie in the following: First, the first visual trajectory and the first sparse point cloud are processed based on the target space transformation parameters between the first visual trajectory and the metric trajectory to obtain a second visual trajectory and a second sparse point cloud with scale. Then, a 3D Gaussian splashing process is applied to the second visual trajectory and the second sparse point cloud to obtain a 3D Gaussian splashing model with scale. This not only ensures that the 3D Gaussian splashing model of the target scene has scale, but also allows this direct 3D Gaussian splashing model to be used in scenarios requiring a true physical scale, such as engineering surveying, geometric deviation analysis, and construction progress monitoring. Furthermore, obtaining the second visual trajectory and the second sparse point cloud with scale before performing the 3D Gaussian splashing model allows for integration with any 3D Gaussian splashing algorithm to achieve 3D Gaussian splashing reconstruction, thereby improving the versatility of the method. Attached Figure Description

[0066] Figure 1 The first flowchart of a three-dimensional Gaussian splashing method based on cross-modal trajectory constraints provided in Embodiment 1 of this disclosure;

[0067] Figure 2The second flowchart of a three-dimensional Gaussian splashing method based on cross-modal trajectory constraints provided in Embodiment 1 of this disclosure;

[0068] Figure 3 The third flowchart of a three-dimensional Gaussian splashing method based on cross-modal trajectory constraints provided in Embodiment 1 of this disclosure;

[0069] Figure 4 A schematic diagram of the alignment process of a three-dimensional Gaussian splashing method based on cross-modal trajectory constraints provided in Embodiment 1 of this disclosure;

[0070] Figure 5 This is a schematic diagram of a three-dimensional Gaussian splashing system based on cross-modal trajectory constraints provided in Embodiment 2 of this disclosure;

[0071] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of this disclosure. Detailed Implementation

[0072] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0073] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the context of the embodiments, and the use of such prefixes should not constitute unnecessary restrictions. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0074] Example 1

[0075] To enable the three-dimensional Gaussian splash model of the target scene to have scale, Embodiment 1 of this disclosure provides a three-dimensional Gaussian splash method based on cross-modal trajectory constraints. Figure 1 This is a first flowchart of a three-dimensional Gaussian splashing method based on cross-modal trajectory constraints provided in Embodiment 1 of this disclosure. The three-dimensional Gaussian splashing method based on cross-modal trajectory constraints includes the following steps:

[0076] Step 101: Obtain the scale-free first visual trajectory, the scale-free first sparse point cloud, and the scaled metric trajectory that correspond to each other in the target scene.

[0077] The first visual trajectory represents the motion path of the image acquisition device in the target scene. The metric trajectory represents the motion path of the scale acquisition device in the target scene.

[0078] Preferably, an integrated device combining image acquisition and scale acquisition can be used to acquire image and scale information of the target scene. Acquiring data through this integrated device allows for more accurate alignment of the subsequent first visual trajectory and measurement trajectory in the time dimension. The image acquisition device can be a visual camera; the scale acquisition device can be a LiDAR, with the LiDAR's odometry outputting a pose sequence with absolute scale to form the measurement trajectory; it can also be an inertial measurement unit, with the inertial odometry integrating to output the measurement trajectory. The scale acquisition device can also be a device using carrier phase differential technology to acquire scale, providing a measurement trajectory with absolute geographic coordinates. The scale acquisition device can also be an ultra-wideband radar, which outputs the measurement trajectory based on anchor point ranging. Any or a combination of the above-mentioned measurement trajectories can be used.

[0079] Step 102: Calculate the target space transformation parameters between the first visual trajectory and the measurement trajectory. The target space transformation parameters are used to align the first visual trajectory and the measurement trajectory in spatial dimensions.

[0080] Step 103: Based on the target space transformation parameters, process the first visual trajectory and the first sparse point cloud to obtain a second visual trajectory and a second sparse point cloud with scale.

[0081] The target space transformation parameters include target scale parameters, target rotation parameters, and target translation parameters.

[0082] The target space transformation parameters are uniformly applied to all pose points on the first visual trajectory and all points on the first sparse point cloud using the following formula:

[0083] ;

[0084] in, For the target scale parameter, For the target rotation parameters, Let p be the target translation parameter, and p be any point on the first visual trajectory or the first sparse point cloud. This represents the coordinates of the point in the physical metric coordinate system after transformation. This yields the second visual trajectory and the second sparse point cloud with scale.

[0085] Furthermore, this transformation only applies to the first visual trajectory and the first sparse point cloud, without modifying any 3D Gaussian splashing algorithm or affecting the trained Gaussian units; this decouples scale recovery from the training of the specific 3D Gaussian splashing algorithm.

[0086] Step 104: Perform 3D Gaussian splashing on the second visual trajectory and the second sparse point cloud to obtain a 3D Gaussian splashing model with scale.

[0087] In this embodiment, the first visual trajectory and the first sparse point cloud are first processed based on the target space transformation parameters between the first visual trajectory and the metric trajectory to obtain a second visual trajectory and a second sparse point cloud with scale. Then, a 3D Gaussian splashing process is applied to the second visual trajectory and the second sparse point cloud to obtain a 3D Gaussian splashing model with scale. This not only gives the 3D Gaussian splashing model of the target scene scale, but also allows it to be directly used in scenarios requiring a realistic physical scale, such as engineering surveying, geometric deviation analysis, and construction progress monitoring. Moreover, obtaining the second visual trajectory and the second sparse point cloud with scale before performing the 3D Gaussian splashing process enables the application of any 3D Gaussian splashing algorithm to achieve 3D Gaussian splashing reconstruction, thereby improving the versatility of the method.

[0088] In one embodiment, the steps of obtaining the scale-free first visual trajectory, the scale-free first sparse point cloud, and the scaled metric trajectory corresponding to each other in the target scene include:

[0089] S2-1: Acquire a sequence of scene images of the target scene.

[0090] S2-2: Process scene image sequences based on sparse reconstruction algorithm to obtain the first visual trajectory and the first sparse point cloud.

[0091] The sparse reconstruction algorithm can be the SfM (Structure from Motion) algorithm.

[0092] In this embodiment, the scene image sequence is processed based on the sparse reconstruction algorithm to obtain the first visual trajectory and the first sparse point cloud, which can provide a basis for subsequent trajectory correspondence and scale recovery.

[0093] In one embodiment, prior to the step of calculating the target space transformation parameters between the first visual trajectory and the metric trajectory, the following is included:

[0094] S3-1: Obtain the timestamps of the first visual trajectory and the metric trajectory respectively.

[0095] S3-2: Align the first visual trajectory and the metric trajectory in the time dimension based on timestamps.

[0096] The first visual trajectory consists of first pose points from several image acquisition devices, and the measurement trajectory consists of second pose points from several measurement acquisition devices. The first and second pose points can be mapped one-to-one based on timestamps to eliminate hardware timing errors between the two acquisition devices. Alternatively, PPS (Pulse Per Second), software timestamp calibration, or time offset compensation can be used to align the first visual trajectory and the measurement trajectory in the time dimension.

[0097] In this embodiment, since timestamp-based alignment has the advantages of low computational overhead and no introduction of additional accumulated errors, directly aligning the first visual trajectory and the metric trajectory in the time dimension using timestamps can significantly reduce the implementation complexity of alignment. Furthermore, the timestamp-aligned first visual trajectory and metric trajectory can provide accurate paired inputs for subsequent scale recovery, thereby making the subsequent scale recovery more accurate.

[0098] In one embodiment, the step of calculating the target space transformation parameters between the first visual trajectory and the metric trajectory includes:

[0099] S4-1: Based on the preset duration and preset step size, the aligned first visual trajectory and measurement trajectory are divided into several time windows.

[0100] S4-2: In response to any time window, if the consistency deviation between the local relative motion of the first visual trajectory and the local relative motion of the measurement trajectory is greater than or equal to the deviation threshold, the first visual trajectory and the measurement trajectory within the corresponding time window are removed to obtain the removed first visual trajectory and the removed measurement trajectory.

[0101] Among them, the local relative motion is characterized by the pose change determined by multiple pose points of the corresponding trajectory within the time window.

[0102] Since the constraints of the corresponding measurement trajectory segment are unreliable when the scale acquisition device degrades in areas lacking geometric features (such as long straight corridors or open halls), the consistency deviation between the local relative motion of the first visual trajectory and the local relative motion of the measurement trajectory is greater than or equal to the deviation threshold. Eliminating the first visual trajectory and measurement trajectory within the corresponding time window can make the remaining first visual trajectory and measurement trajectory more reliable.

[0103] It should be noted that this deviation threshold can be set according to the actual situation.

[0104] S4-3: Calculate the target space transformation parameters based on the first visual trajectory after culling and the metric trajectory after culling.

[0105] In this embodiment, windowing is used to perform local screening of the first visual trajectory and the measurement trajectory to accurately identify locally degraded portions. Furthermore, by setting a deviation threshold, unreliable data segments caused by geometric degradation or temporal lock-out on the first visual trajectory and the measurement trajectory are actively removed, effectively preventing unreliable data segments from affecting the accuracy of the target transformation parameter solution. Finally, the spatial transformation parameters are solved based on the screened accurate trajectories, ensuring that the target spatial parameters are calculated based on high-confidence trajectory data, thereby improving the stability and reliability of the final scale recovery result.

[0106] In one embodiment, the first visual trajectory after the culling process includes several first pose points, and the metric trajectory after the culling process includes several second pose points. Furthermore, the first pose points and the second pose points correspond to each other.

[0107] The steps for calculating the target space transformation parameters between the first visual trajectory and the metric trajectory include:

[0108] S5-1: Based on the first pose point and the second pose point, calculate the initial spatial transformation parameters between the first visual trajectory after culling and the metric trajectory after culling.

[0109] The initial spatial transformation parameters may include initial scale parameters, initial rotation parameters, and initial translation parameters.

[0110] The specific calculation method can be as follows: Taking five pairs of first pose points and second pose points on the first visual trajectory and the metric trajectory as an example:

[0111] The calculation process for the initial scale parameter is as follows: calculate the span of the first visual trajectory based on the first pose point; calculate the span of the metric trajectory based on the second pose point, and the initial scale parameter is the span ratio between the two.

[0112] The calculation process for the initial rotation parameters is as follows: calculate the centroids of the two trajectories based on the first pose point and the second pose point respectively, remove the centroids, and then solve for the initial rotation parameters by the singular value decomposition of the weighted cross covariance matrix.

[0113] The calculation process for the initial translation parameters is as follows: calculate the centroids of the two trajectories based on the first pose point and the second pose point respectively, and calculate the initial translation parameters based on the centroids.

[0114] S5-2: Calculate the target space transformation parameters based on the initial space transformation parameters.

[0115] In this embodiment, the initial spatial transformation parameters are solved based on the first pose point on the first visual trajectory after the removal process and the second pose point on the metric trajectory after the removal process. This provides an initial value close to the true value for subsequent iterative optimization, avoiding divergence or slow convergence in subsequent iterations due to poorly guessed initial spatial transformation parameters. This significantly reduces the number of subsequent iterations and computational overhead.

[0116] In one embodiment, the step of calculating the target spatial transformation parameters based on the initial spatial transformation parameters includes:

[0117] S6-1: Based on the initial spatial transformation parameters, construct the target loss function between the first visual trajectory and the metric trajectory.

[0118] S6-2: Iteratively update the initial spatial transformation parameters based on the preset iterative logic.

[0119] The preset iteration logic is as follows:

[0120] Taking the above 5 pairs of first and second pose points as an example:

[0121] The initial spatial transformation parameters are applied to the first pose point, the residual between the first pose point and the corresponding second pose point is calculated, the weight of each pose point is adjusted based on the residual, and the initial spatial transformation parameters are recalculated based on the pose points with adjusted weights to update the initial spatial transformation parameters.

[0122] The logic for adjusting the weights is as follows:

[0123] ;

[0124] Where x is the squared residual corresponding to the pose point in the current iteration; δ is the residual threshold, which can be adjusted according to the actual time alignment error magnitude and the noise level of the metric sensor. Preferably, the residual threshold is 0.1m. The robust loss between the first visual trajectory and the metric trajectory.

[0125] S6-3: Based on the objective loss function, calculate the loss value corresponding to the initial spatial transformation parameters after each iteration update.

[0126] S6-4: In response to the loss value being less than or equal to the preset loss value, the initial spatial transformation parameter corresponding to the current loss value is used as the target spatial transformation parameter.

[0127] In this embodiment, after defining the optimization objective starting with the initial spatial transformation parameters, the initial spatial transformation parameters are updated through a preset iterative logic. The quality of the initial spatial transformation parameters after each update is quantified using a target loss function, and the convergence criterion is whether the loss value meets the target. This makes the calculated target spatial transformation parameters more accurate, thereby improving the accuracy of the final scale recovery result.

[0128] In one embodiment, the initial spatial transformation parameters include initial scale parameters, initial rotation parameters, and initial translation parameters.

[0129] The target loss function is:

[0130] .

[0131] Where ρ(·) is the robust loss between the first visual trajectory and the metric trajectory. The first visual trajectory. To measure the trajectory. These are the initial scale parameters. These are the initial rotation parameters. represents the initial translation parameters. N is the total number of pose pairs formed by the first pose point and the second pose point. i∈[1,N].

[0132] In this embodiment, the target loss function can make the calculated target space transformation parameters more accurate, thereby improving the accuracy of the final scale recovery result.

[0133] In one embodiment, Figure 2 This is a second flowchart of a three-dimensional Gaussian splashing method based on cross-modal trajectory constraints provided in Embodiment 1 of this disclosure; combined with Figure 2 The three-dimensional Gaussian splashing method based on cross-modal trajectory constraints is further explained.

[0134] S7-1: Data is collected based on the data acquisition module.

[0135] Specifically, a visual camera (the image acquisition device mentioned above) and a measurement sensor (the scale acquisition device mentioned above) are used, and the two data streams are time-aligned, and the image sequence and measurement sensor data are output.

[0136] S7-2: Sparse reconstruction of image sequences using the sparse reconstruction module.

[0137] Specifically, taking the image sequence as input, SfM sparse reconstruction is performed, and the output is a scale-free camera pose sequence (the first visual trajectory mentioned above) and a scale-free sparse point cloud (the first sparse point cloud mentioned above).

[0138] S7-3: Processes measurement sensor data based on the measurement trajectory module.

[0139] Specifically, it receives measurement sensor data and outputs a measurement trajectory with absolute scale.

[0140] S7-4: The trajectory correspondence module will align the camera pose sequence and the metric trajectory.

[0141] Specifically, based on temporal alignment, the visual trajectory and the metric trajectory are paired pose by pose to align the camera pose sequence and the metric trajectory in the temporal dimension.

[0142] S7-5: Based on the trajectory filtering module, identify and remove geometrically degenerate segments in the metric trajectory, and output the set of pose pairs after removal (i.e., the first visual trajectory and the metric trajectory after removal in the above process).

[0143] S7-6: Calculate the target space transformation parameters based on the global scale recovery module.

[0144] Specifically, a robust optimization model is constructed on the set of pose pairs after the removal process to solve for the target space transformation parameters that achieve global optimal alignment between the two trajectories. Initial scale parameters are calculated based on the scale submodule, initial rotation parameters are calculated based on the rotation submodule, and initial translation parameters are calculated based on the translation submodule. The weights of the pose pairs after the removal process are then updated according to the weight update module.

[0145] S7-7: Based on the transformation application module, the converged target space transformation parameters are applied to all scale-free camera pose sequences and scale-free sparse point clouds at once to obtain scaled camera pose sequences (i.e., the second visual trajectory mentioned above) and scaled second sparse point clouds.

[0146] S7-8: Based on the 3D Gaussian splashing module, the camera pose sequence with scale and the second sparse point cloud with scale are processed to obtain a 3D Gaussian splashing model with metric scale.

[0147] In one embodiment, Figure 3 This is a third flowchart of a three-dimensional Gaussian splashing method based on cross-modal trajectory constraints provided in Embodiment 1 of this disclosure; combined with Figure 3 The three-dimensional Gaussian splashing method based on cross-modal trajectory constraints is further explained.

[0148] S8-1: Data acquisition and time alignment.

[0149] The target scene is captured by a visual camera and a measurement sensor, and the data from the two sources are aligned based on timestamps.

[0150] S8-2: Process the image sequence based on sparse reconstruction to obtain scale-free camera pose and scale-free sparse point cloud.

[0151] S8-3: Processes the measurement sensor data to generate the measurement trajectory.

[0152] S8-4: Establishing the first visual trajectory based on scale-free camera pose.

[0153] S8-5: Identify and remove degenerate trajectory segments to obtain the first visual trajectory and the metric trajectory after removal.

[0154] S8-6: Construct a robust optimization model based on the first visual trajectory after removal and the metric trajectory after removal, and solve for the initial spatial transformation parameters containing the scale parameter.

[0155] S8-7: Iteratively update the initial space transformation parameters until convergence to obtain the target space transformation parameters.

[0156] S8-8: Apply the target space transformation parameters to all scale-free camera poses and sparse point clouds at once to obtain metric-scale poses and sparse point clouds.

[0157] S8-9: Using metric-scale pose and sparse point cloud as input, a metric-scale 3D Gaussian splash model is trained.

[0158] In one embodiment, Figure 4 This is a schematic diagram illustrating the alignment process of a three-dimensional Gaussian splashing method based on cross-modal trajectory constraints provided in Embodiment 1 of this disclosure. The first visual trajectory 41 is processed by spatial transformation parameters and then overlapped with the metric trajectory 42 to obtain an overlapping trajectory 43. Figure 4 It can be clearly seen that the trajectory error between the processed first visual trajectory 41 and the measurement trajectory 42 is very small and can be ignored.

[0159] In one embodiment, taking a real indoor scene as the target scene as an example, this article explains how to obtain a scaled three-dimensional Gaussian splash model of the target scene.

[0160] The first step is to collect the scene and configuration.

[0161] Data was collected in an indoor scene of approximately 300 m², encompassing various reconstruction challenges, including sparsely characterized areas, heavily occluded and cluttered areas, areas with reflective glass / metal surfaces, and regular structural areas with standardized geometric references (badminton court markings). This allowed for thorough testing of the effectiveness of scale restoration under complex indoor conditions. The data acquisition device was an integrated handheld multi-sensor unit, integrating three industrial-grade fisheye cameras, a LiDAR, and an RTK (Real-Time Kinematic) module; hardware synchronization was maintained between the cameras and the LiDAR. Each data acquisition session lasted approximately 14 minutes.

[0162] The second step is to analyze the scale restoration results.

[0163] Along a complete acquisition path of approximately 600 m, the image sequence acquired by the camera and the measurement trajectory acquired by the LiDAR yielded a global scale factor of approximately 1.94. This global scale factor is the target scale parameter in the target space transformation parameters. By uniformly adjusting the originally dimensionless monocular motion reconstruction results to the real physical scale, it is proven that this method performs real-scale restoration, rather than the model itself having a pre-existing scale. Re-estimating the global scale factor along the trajectory using a sliding window, its coefficient of variation across 29 independent sliding windows is approximately 0.89%, indicating that the restored scale is independent of specific trajectory segments and highly consistent. The aligned first visual trajectory closely matches the measurement trajectory, with residual differences primarily representing a single global scale rather than segment-by-segment drift.

[0164] The third step is to verify the measurement accuracy.

[0165] Using lidar point clouds as the ground truth, the geometry of real components was directly measured using a 3D Gaussian splash model trained with the reconstructed scale obtained by this method. Measurements were performed on badminton court landmarks with standard dimensions; the standard doubles court width is 6.10 m, the measured width is 6.07 m, and the relative error for the full width measurement is approximately 0.5%. The average absolute error across multiple types of building components was 1.71 cm. The median point-to-point distance between the reconstructed point cloud and the ground truth lidar point cloud was 1.87 cm. These accuracies meet the centimeter-level tolerance requirements for engineering surveying and construction monitoring.

[0166] In this embodiment, the present disclosure utilizes the correspondence between a scale-free first visual trajectory and a metric trajectory to generate target space variation parameters through global trajectory optimization, thereby obtaining global scale transformation parameters. These global scale transformation parameters are then uniformly applied to the scale-free first visual trajectory and the first sparse point cloud to generate a scaled second visual trajectory and a second sparse point cloud. The second visual trajectory and the second sparse point cloud are then directly used as the metric scale input for the 3D Gaussian splashing algorithm to obtain a scaled 3D Gaussian splashing model. Thus, the 3D Gaussian splashing method of this disclosure can recover absolute scale; it is decoupled from the 3D Gaussian splashing algorithm; it can be adapted to different 3D Gaussian splashing algorithm implementations; and the metric source is scalable.

[0167] This disclosure also identifies and eliminates unreliable trajectory segments caused by geometric degradation based on cross-modal relative pose consistency and metric odometry observability, constructing a reliable pose correspondence set for global scale recovery. This improves scale estimation stability and reduces the impact of degraded scenes on the global scale.

[0168] Furthermore, this disclosure also suppresses anomalous matching / outlier correspondences and improves scale recovery accuracy through a robust global trajectory optimization mechanism. Further, this disclosure takes the entire first visual trajectory and sparse point cloud after sparse reconstruction as the object, and after sparse reconstruction and before the 3D Gaussian splashing algorithm processing, establishes a correspondence between the first visual trajectory and the metric trajectory and performs global, one-time trajectory-level scale recovery. This avoids the problem of sensitivity to cumulative drift caused by the incremental process of relying on each pair of adjacent frame point clouds in point-by-point recovery.

[0169] Example 2

[0170] Corresponding to the aforementioned embodiment 1 of the three-dimensional Gaussian splashing method based on cross-modal trajectory constraints, this disclosure also provides an embodiment of a three-dimensional Gaussian splashing system based on cross-modal trajectory constraints. Figure 5 This is a schematic diagram of a three-dimensional Gaussian splashing system based on cross-modal trajectory constraints provided in Embodiment 2 of this disclosure. The three-dimensional Gaussian splashing system 50 based on cross-modal trajectory constraints includes:

[0171] The acquisition module 51 is used to acquire the scale-free first visual trajectory, the scale-free first sparse point cloud, and the scaled metric trajectory that correspond to each other in the target scene; wherein, the first visual trajectory is used to represent the motion path of the image acquisition device in the target scene; and the metric trajectory is used to represent the motion path of the scale acquisition device in the target scene.

[0172] Calculation module 52 is used to calculate the target space transformation parameters between the first visual trajectory and the measurement trajectory; the target space transformation parameters are used to align the first visual trajectory and the measurement trajectory in spatial dimensions;

[0173] The processing module 53 is used to process the first visual trajectory and the first sparse point cloud based on the target space transformation parameters to obtain a second visual trajectory and a second sparse point cloud with scale.

[0174] The processing module 53 is also used to perform three-dimensional Gaussian splashing processing on the second visual trajectory and the second sparse point cloud to obtain a three-dimensional Gaussian splashing model with scale.

[0175] In one embodiment, the acquisition module 51 includes:

[0176] The acquisition unit is used to acquire scene image sequences of the target scene;

[0177] The processing unit is used to process the scene image sequence based on the sparse reconstruction algorithm to obtain the first visual trajectory and the first sparse point cloud.

[0178] In one embodiment, the acquisition module 51 is further configured to acquire the timestamps of the first visual trajectory and the measurement trajectory, respectively.

[0179] The three-dimensional Gaussian splash system also includes:

[0180] The alignment module aligns the first visual trajectory and the metric trajectory along the time dimension, based on the timestamp.

[0181] In one embodiment, the computing module 52 includes:

[0182] The segmentation unit is used to divide the aligned first visual trajectory and measurement trajectory into several time windows based on a preset duration and a preset step size.

[0183] The response unit, in response to any time window where the consistency deviation between the local relative motion of the first visual trajectory and the local relative motion of the measurement trajectory is greater than or equal to a deviation threshold, removes the first visual trajectory and the measurement trajectory within the corresponding time window to obtain the first visual trajectory and the measurement trajectory after removal processing; wherein, the local relative motion represents the pose change amount jointly determined by multiple pose points of the trajectory and its corresponding trajectory within the time window.

[0184] The calculation module 52 is also used to calculate the target space transformation parameters based on the first visual trajectory after culling and the metric trajectory after culling.

[0185] In one embodiment, the first visual trajectory after the removal process includes several first pose points, and the measurement trajectory after the removal process includes several second pose points; and the first pose points and the second pose points correspond to each other.

[0186] The calculation module 52 also calculates the initial spatial transformation parameters between the first visual trajectory after culling and the metric trajectory after culling, based on the first pose point and the second pose point.

[0187] The calculation module 52 also calculates the target space transformation parameters based on the initial space transformation parameters.

[0188] In one embodiment, the calculation module 52 further includes:

[0189] The building unit is used to construct the target loss function between the first visual trajectory and the metric trajectory based on the initial spatial transformation parameters;

[0190] The update unit is used to iteratively update the initial spatial transformation parameters based on a preset iterative logic;

[0191] The calculation module 52 also calculates the loss value corresponding to the initial spatial transformation parameters after each iteration update based on the target loss function;

[0192] The response unit also responds to the loss value being less than or equal to the preset loss value by using the initial spatial transformation parameter corresponding to the current loss value as the target spatial transformation parameter.

[0193] In one embodiment, the initial spatial transformation parameters include initial scale parameters, initial rotation parameters, and initial translation parameters;

[0194] The target loss function is:

[0195] ;

[0196] Where ρ(·) is the robust loss between the first visual trajectory and the metric trajectory; The first visual trajectory; To measure the trajectory; These are the initial scale parameters; These are the initial rotation parameters; is the initial translation parameter; N is the total number of pose pairs formed by the first pose point and the second pose point; i∈[1,N].

[0197] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.

[0198] Example 3

[0199] Figure 6 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the three-dimensional Gaussian splashing method based on cross-modal trajectory constraints as described in any of the above embodiments. Figure 6 The electronic device 60 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0200] like Figure 6 As shown, the electronic device 60 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 60 may include, but are not limited to: at least one processor 61, at least one memory 62, and a bus 63 connecting different system components (including memory 62 and processor 61).

[0201] Bus 63 includes a data bus, an address bus, and a control bus.

[0202] The memory 62 may include volatile memory, such as random access memory (RAM) 621 and / or cache memory 622, and may further include read-only memory (ROM) 623.

[0203] The memory 62 may also include a program tool 625 (or utility) having a set (at least one) program module 624, such program module 624 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0204] The processor 61 executes various functional applications and data processing by running computer programs stored in the memory 62, such as the three-dimensional Gaussian splashing method based on cross-modal trajectory constraints provided in any of the above embodiments.

[0205] Electronic device 60 can also communicate with one or more external devices 64 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 65. Furthermore, electronic device 60 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 66. As shown, network adapter 66 communicates with other modules of electronic device 60 via bus 63. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 60, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0206] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0207] Example 4

[0208] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the three-dimensional Gaussian splashing method based on cross-modal trajectory constraints provided in any of the above embodiments.

[0209] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0210] Example 5

[0211] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the three-dimensional Gaussian splashing method based on cross-modal trajectory constraints as described above.

[0212] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0213] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A three-dimensional Gaussian splashing method based on cross-modal trajectory constraints, characterized in that, The three-dimensional Gaussian splashing method includes: The system acquires a scale-free first visual trajectory, a scale-free first sparse point cloud, and a scaled metric trajectory that correspond to each other in the target scene. The first visual trajectory is used to characterize the motion path of the image acquisition device in the target scene, and the metric trajectory is used to characterize the motion path of the scale acquisition device in the target scene. Calculate the target space transformation parameters between the first visual trajectory and the measurement trajectory; the target space transformation parameters are used to align the first visual trajectory and the measurement trajectory in spatial dimensions; Based on the target space transformation parameters, the first visual trajectory and the first sparse point cloud are processed to obtain a second visual trajectory and a second sparse point cloud with scale. The second visual trajectory and the second sparse point cloud are subjected to three-dimensional Gaussian splashing to obtain a three-dimensional Gaussian splashing model with scale.

2. The three-dimensional Gaussian splashing method as described in claim 1, characterized in that, The steps for obtaining the scale-free first visual trajectory, the scale-free first sparse point cloud, and the scaled metric trajectory corresponding to each other in the target scene include: Acquire a sequence of scene images of the target scene; The scene image sequence is processed based on a sparse reconstruction algorithm to obtain the first visual trajectory and the first sparse point cloud.

3. The three-dimensional Gaussian splashing method as described in claim 1, characterized in that, Prior to the step of calculating the target space transformation parameters between the first visual trajectory and the metric trajectory, the following is included: Obtain the timestamps of the first visual trajectory and the measurement trajectory respectively; Based on the timestamp, the first visual trajectory and the metric trajectory are aligned in the time dimension.

4. The three-dimensional Gaussian splashing method as described in claim 3, characterized in that, The step of calculating the target space transformation parameters between the first visual trajectory and the metric trajectory includes: Based on a preset duration and a preset step size, the aligned first visual trajectory and the measurement trajectory are divided into several time windows; In response to any time window, if the consistency deviation between the local relative motion of the first visual trajectory and the local relative motion of the measurement trajectory is greater than or equal to the deviation threshold, the first visual trajectory and the measurement trajectory within the corresponding time window are removed to obtain the first visual trajectory and the measurement trajectory after removal processing; wherein, the local relative motion represents the pose change amount jointly determined by multiple pose points of the trajectory and its corresponding trajectory within the time window. The target space transformation parameters are calculated based on the first visual trajectory after the removal process and the metric trajectory after the removal process.

5. The three-dimensional Gaussian splashing method as described in claim 4, characterized in that, The first visual trajectory after the removal process includes several first pose points, and the metric trajectory after the removal process includes several second pose points; and the first pose points and the second pose points correspond to each other; The step of calculating the target space transformation parameters between the first visual trajectory and the metric trajectory includes: Based on the first pose point and the second pose point, calculate the initial spatial transformation parameters between the first visual trajectory after culling and the metric trajectory after culling. The target spatial transformation parameters are calculated based on the initial spatial transformation parameters.

6. The three-dimensional Gaussian splashing method as described in claim 5, characterized in that, The step of calculating the target spatial transformation parameters based on the initial spatial transformation parameters includes: Based on the initial spatial transformation parameters, a target loss function is constructed between the first visual trajectory and the metric trajectory; The initial spatial transformation parameters are iteratively updated based on a preset iterative logic. Based on the target loss function, calculate the loss value corresponding to the initial spatial transformation parameters after each iteration update; In response to the loss value being less than or equal to a preset loss value, the initial spatial transformation parameter corresponding to the current loss value is used as the target spatial transformation parameter.

7. The three-dimensional Gaussian splashing method as described in claim 6, characterized in that, The initial spatial transformation parameters include initial scale parameters, initial rotation parameters, and initial translation parameters; The target loss function is: ； Wherein, ρ(·) is the robust loss between the first visual trajectory and the metric trajectory; The first visual trajectory; For the metric trajectory; The initial scale parameter; The initial rotation parameters are... The initial translation parameter is denoted as ; N is the total number of pose pairs formed by the first pose point and the second pose point; i∈[1,N].

8. A three-dimensional Gaussian splashing system based on cross-modal trajectory constraints, characterized in that, The three-dimensional Gaussian splashing system includes: The acquisition module is used to acquire a scale-free first visual trajectory, a scale-free first sparse point cloud, and a scaled metric trajectory that correspond to each other in the target scene; wherein, the first visual trajectory is used to characterize the motion path of the image acquisition device in the target scene; and the metric trajectory is used to characterize the motion path of the scale acquisition device in the target scene. The calculation module is used to calculate the target space transformation parameters between the first visual trajectory and the measurement trajectory; the target space transformation parameters are used to align the first visual trajectory and the measurement trajectory in spatial dimensions. The processing module is used to process the first visual trajectory and the first sparse point cloud based on the target space transformation parameters to obtain a second visual trajectory and a second sparse point cloud with scale. The processing module is also used to perform three-dimensional Gaussian splashing processing on the second visual trajectory and the second sparse point cloud to obtain a three-dimensional Gaussian splashing model with scale.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the three-dimensional Gaussian splashing method based on cross-modal trajectory constraints as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional Gaussian splashing method based on cross-modal trajectory constraints as described in any one of claims 1 to 7.

Images