Improved method for three-dimensional reconstruction of gaussian sputtering based on pose prior

Abstract

The application provides a Gaussian sputtering three-dimensional reconstruction improvement method based on pose prior, relates to the field of Gaussian sputtering three-dimensional reconstruction, and comprises the following steps: collecting multi-modal information by using a MarsLogger mobile phone application, wherein the multi-modal information comprises video data and inertial measurement unit data; estimating pose information at a corresponding time point according to the inertial measurement unit data; converting the video data into pictures by frame extraction, accurately matching the pictures with the pose by using a three-dimensional linear interpolation method and a three-dimensional spherical interpolation method according to timestamps, so as to obtain matched picture poses; performing COLMAP sparse reconstruction based on the pose prior according to the matched picture poses, so as to obtain a sparse reconstruction point cloud; and performing 3DGS reconstruction according to the sparse reconstruction point cloud. The application aims to solve the problem that the reconstruction effect of the prior art is poor in a complex environment (such as a mine tunnel).

Description

TECHNICAL FIELD

[0001] The application relates to the field of Gaussian sputtering three-dimensional reconstruction, in particular to a Gaussian sputtering three-dimensional reconstruction improvement method based on pose prior. BACKGROUND

[0002] In recent years, with the development of computer graphics and computer vision technology, three-dimensional reconstruction and novel view synthesis technology have attracted widespread attention. Among them, NeRF (Neural Radiance Field) and 3D Gaussian Splatting (3DGS) are two representative technologies.

[0003] NeRF can realize high-quality novel view synthesis by optimizing the neural network to represent the volume density and color information of the scene. However, the training and rendering process of NeRF usually requires a large amount of time and computing resources, making it difficult to meet real-time requirements. In addition, NeRF is prone to overfitting and loss of details when dealing with complex scenes.

[0004] In contrast, 3DGS has significant advantages. It uses a representation method based on 3D Gaussian distribution, which can efficiently render and synthesize novel views. The rendering speed of 3DGS is much faster than that of NeRF, which can provide high-quality visual effects in real-time applications. In addition, 3DGS has better flexibility and scalability when representing complex scenes, which can better capture the details and structure of the scene.

[0005] However, the input of 3DGS usually depends on the sparse point cloud generated by the Structure from Motion (SfM) algorithm (such as COLMAP). In complex environments such as mine tunnels, the feature extraction and sparse point cloud reconstruction of COLMAP are often not ideal due to poor lighting conditions and single texture features. This is mainly because the mine tunnel environment lacks sufficient texture information, making it difficult to accurately detect and match feature points. In addition, the complex geometric structure and occlusion phenomenon in the mine tunnel further increase the difficulty of feature extraction. This suboptimal sparse point cloud reconstruction result directly affects the three-dimensional reconstruction effect of 3DGS, resulting in errors and incompleteness in the reconstructed three-dimensional model.

[0006] Therefore, how to improve the sparse point cloud reconstruction quality of COLMAP in complex environments such as mine tunnels to provide more accurate input for 3DGS is one of the problems to be solved in the field of three-dimensional reconstruction. The present application proposes a 3DGS three-dimensional reconstruction improvement method based on pose prior, which introduces pose prior information to optimize the sparse point cloud reconstruction process of COLMAP, thereby improving the three-dimensional reconstruction effect of 3DGS. SUMMARY

[0007] The application provides a Gaussian sputtering three-dimensional reconstruction improved method based on pose prior, to solve the problem of poor reconstruction effect in complex environment (such as mine tunnel) in the prior art.

[0008] In one aspect, the application provides a Gaussian sputtering three-dimensional reconstruction improved method based on pose prior, comprising:

[0009] The multi-modal information is collected by using a MarsLogger mobile phone application, and the multi-modal information includes video data and inertial measurement unit data;

[0010] According to the inertial measurement unit data, the pose information at the corresponding time point is estimated;

[0011] The video data is frame-extracted and converted into pictures, and the pictures are accurately matched with the poses by using three-dimensional linear interpolation and three-dimensional spherical interpolation methods according to the time stamps, so as to obtain the matched picture poses;

[0012] According to the matched picture poses, COLMAP sparse reconstruction based on pose prior is performed, to obtain sparse reconstruction point clouds;

[0013] According to the sparse reconstruction point clouds, 3DGS reconstruction is performed.

[0014] Further, according to the inertial measurement unit data, the pose information at the corresponding time point is estimated, comprising:

[0015] The inertial measurement unit data is obtained and preprocessed to obtain a preprocessed inertial measurement unit data set, and the inertial measurement unit data includes a time stamp sequence, three-axis angular velocity of a gyroscope in an inertial measurement unit self-coordinate system and three-axis acceleration of an accelerometer;

[0016] According to the inertial measurement unit data set, the acceleration data in a stationary state is analyzed to detect the stationary state of the inertial measurement unit, to obtain a stationary state detection result;

[0017] According to the stationary state detection result, the gravity acceleration measured by the accelerometer in the stationary state is analyzed, and the initial attitude of the inertial measurement unit is calculated;

[0018] During the movement, the initial attitude of the inertial measurement unit is updated by using the gyroscope and accelerometer data, and the gain K of the acceleration data is set to 0 in the stationary state P , and the gain K is set to 1 in the non-stationary state P , to obtain the updated initial attitude of the inertial measurement unit;

[0019] According to the updated initial attitude of the inertial measurement unit, the velocity and displacement relative to the initial position in the earth coordinate system are calculated, to obtain the reconstructed movement trajectory, i.e., the pose information at the corresponding time point.

[0020] Further, according to the inertial measurement unit data set, the acceleration data in the stationary state is analyzed to detect the stationary state of the inertial measurement unit to obtain a stationary state detection result, including:

[0021] According to the inertial measurement unit data set, the acceleration amplitude is calculated;

[0022] According to the acceleration amplitude, a cutoff frequency is set to remove the gravitational acceleration and static acceleration component to obtain a high-pass filtered acceleration amplitude;

[0023] According to the high-pass filtered acceleration amplitude, the absolute value is taken and low-pass filtered to obtain a low-pass filtered acceleration amplitude;

[0024] According to the low-pass filtered acceleration amplitude, threshold detection is performed, if the acceleration amplitude is less than the threshold, it is marked as a stationary state, if the acceleration amplitude is not less than the threshold, it is marked as a motion state, to obtain a stationary state detection result;

[0025] At the moment when the motion state is detected, the state change in the preset time range before and after is checked, and the edge value is set to smooth the switching between the stationary state and the motion state.

[0026] Further, according to the stationary state detection result, the gravitational acceleration measured by the accelerometer in the stationary state is analyzed, and the initial attitude of the inertial measurement unit is calculated, including:

[0027] According to the stationary state detection result, the initial angle is assumed to be 0, and the quaternion is represented by Quaternion=q=[1,0,0,0], wherein, Quaternion is represents the quaternion of the sensor relative to the earth, q is represents the quaternion of the earth relative to the sensor, according to the conjugate relationship of the quaternion, Quaternion=q * =[q 0’ -q 1’ -q 2’ -q3], wherein q0 is the real part, q1, q2, q3 are the imaginary parts;

[0028] According to the stationary state detection result, the time window of the initial stationary state of the inertial measurement unit is selected, the average value of the acceleration vector measured in the stationary state in the window is calculated and normalized to obtain a unitized acceleration, wherein the acceleration measured in the stationary state is The unitized acceleration is represents the unit component of the gravitational direction in the inertial measurement unit coordinate system, a x , a y , az are accelerations in x, y, z directions in the IMU coordinate system;

[0029] According to the quaternion and the unitized acceleration, when the carrier is stationary, the initial attitude of the IMU is estimated using The relationship between the accelerometer output and the gravitational acceleration is calculated, and the estimated gravitational direction is used The estimated gravitational direction is obtained, where a x , a y , a z are accelerations in x, y, z directions in the IMU coordinate system, q0 is the real part, q1, q2, q3 are the imaginary parts, is the gravitational direction, g is the gravitational acceleration, is the relationship between the accelerometer output and the gravitational acceleration;

[0030] According to the gravitational direction of the unitized acceleration and the estimated gravitational direction, the included angle error is calculated by the cross product;

[0031] According to the included angle error, the angular velocity of the gyroscope is corrected using InitError = K i ·InitError + K P ·Error, and the updated IMU attitude is used , where IntError is the error integral term, K p and K i represent the calculation error gain and the integral error gain, respectively, and Error is the error, is the updated quaternion, Δt is the time interval, and t is the time, represents the derivative of the quaternion, and the rate of change of the current attitude quaternion is calculated using quaternion multiplication: represents quaternion multiplication, s w t is the constructed angular velocity quaternion;

[0032] The above steps are iteratively performed until the error between the estimated gravitational direction and the actual gravitational direction is minimized to obtain an accurate initial attitude estimate, i.e. the initial attitude of the IMU.

[0033] Further, according to the updated initial attitude of the IMU, the velocity and displacement relative to the initial position in the earth coordinate system are calculated to obtain the reconstructed motion trajectory, i.e. the pose information at the corresponding time point, including:

[0034] According to the updated initial attitude of the IMU, the acceleration measured by the IMU is converted into the earth reference system using to obtain the acceleration in the earth reference system, where, Ea represents the acceleration in the earth reference frame, S a represents the acceleration in the inertial measurement unit reference frame, is an updated quaternion, represents a quaternion multiplication, represents a quaternion in the earth reference frame;

[0035] According to the acceleration in the earth reference frame, the velocity in the earth coordinate system is calculated using v t = v t-1 + E a t · Δt, wherein, E a t represents the acceleration at time t, v t represents the velocity at time t, and Δt is the time interval;

[0036] According to the velocity in the earth coordinate system, the velocity drift is calculated using linear interpolation combined with the rate of change of velocity and the period of motion, and the velocity drift is subtracted from the velocity in the earth coordinate system to obtain the corrected velocity;

[0037] According to the corrected velocity, the position matrix is initialized to obtain the reconstructed motion trajectory, i.e., the pose information corresponding to the time point.

[0038] Further, the video data is frame-extracted into pictures, and the pictures and poses are accurately matched using three-dimensional linear interpolation and three-dimensional spherical interpolation methods according to the timestamps to obtain matched picture poses, including:

[0039] The video data is frame-extracted into pictures to obtain a list of sampled video timestamps;

[0040] According to the list of sampled video timestamps, video frame timestamps that are not within the range of the inertial measurement unit timestamps are removed to obtain a list of valid video frame timestamps;

[0041] According to the list of valid video frame timestamps, the matched picture poses are obtained based on the positions and poses of the inertial measurement units at adjacent time points, wherein the positions are achieved by three-dimensional linear interpolation, and the poses are achieved by quaternion interpolation, and the quaternion interpolation adopts a spherical linear interpolation method, and the formula is: wherein s is an interpolation factor, q0 is the inertial measurement unit pose corresponding to the adjacent previous time stamp, q1 is the inertial measurement unit pose corresponding to the adjacent next time stamp, and θ is the included angle between q0 and q1.

[0042] Further, according to the matched picture poses, COLMAP sparse reconstruction based on pose priors is performed to obtain a sparse reconstruction point cloud, including:

[0043] Based on the pose of the matched image, key points and descriptors are extracted from the input image using a detection algorithm. Corresponding feature point pairs are matched based on the similarity of the descriptors and geometric verification is performed to obtain the matched feature point pairs.

[0044] Based on the matched feature point pairs, sort all images, select the two images with the most matched feature point pairs to initialize, and obtain the initialized image;

[0045] Based on the initialized image, select the image with the most matching points or the widest distribution of visible points, and adjust and optimize the pose of the new image to obtain the incrementally reconstructed image;

[0046] Repeat the above steps to optimize all generated 3D points and camera poses as a whole, and filter and remove from the optimized point cloud and image to obtain a sparse reconstructed point cloud.

[0047] Furthermore, based on the sparsely reconstructed point cloud, 3DGS reconstruction is performed, including:

[0048] Based on the sparsely reconstructed point cloud, COLMAP sparse reconstruction was used to obtain the point cloud and camera parameters, and the point cloud was initialized with a 3D Gaussian distribution, wherein the 3D Gaussian distribution is G. i ={μ i ,Σ i ,α i ,c i}, where μ i It is a position, Σ i It is the covariance matrix, α i It is opacity, c i These are spherical harmonic coefficients;

[0049] Based on a 3D Gaussian distribution, by minimizing the error between the reconstructed image and the training view, using... The parameters of the Gaussian distribution are optimized to obtain the optimized 3D Gaussian distribution parameters, where, It is L1 loss. It is the D-SSIM loss, and λ is the balance parameter;

[0050] In the process of optimizing 3D Gaussian distribution parameters, the number and density of Gaussian distributions are dynamically adjusted through cloning and splitting operations.

[0051] Based on the optimized 3D Gaussian distribution parameters, a tile rasterization algorithm is used for real-time rendering to obtain the rendered 3D Gaussian distribution.

[0052] Repeat the above steps for 3D Gaussian distribution parameter optimization and real-time rendering until the preset number of iterations or convergence conditions are reached.

[0053] Further, in the 3D Gaussian distribution parameter optimization process, the number and density of Gaussian distributions are dynamically adjusted through cloning and splitting operations, including:

[0054] If the position gradient of the small-scale Gaussian distribution is large, it indicates that more geometric details are needed in this area, and the number of Gaussian distributions is increased through the cloning operation:

[0055] And alpha i <theta α , cloning

[0056] If the position gradient of the large-scale Gaussian distribution is large, it indicates that the area needs to be divided more finely, and a large Gaussian distribution is split into two small Gaussian distributions through the splitting operation:

[0057] And alpha i >= theta α , splitting

[0058] Where mu i is the position, alpha i is the opacity, theta pos is the gradient threshold, and theta α is the opacity threshold.

[0059] Further, according to the optimized 3D Gaussian distribution parameters, a tile rasterization algorithm is used for real-time rendering to obtain the rendered 3D Gaussian distribution, including:

[0060] According to the optimized 3D Gaussian distribution parameters, the screen is divided into 16x16 pixel tiles;

[0061] The Gaussian distributions in each tile are sorted in depth using a GPU quicksort algorithm to obtain sorted Gaussian distributions;

[0062] According to the sorted Gaussian distributions, each pixel is rasterized, and the cumulative value of color and opacity is calculated, where p is the pixel position, C(p) is the color value of the pixel, N is the total number of samples, i is the index, mu i is the position, alpha i is the opacity, c i is the spherical harmonic coefficient, and T is the time.

[0063] The improved method for Gaussian sputtering three-dimensional reconstruction based on pose prior provided by the application significantly improves the reconstruction efficiency and accuracy by introducing IMU data as pose prior, Figure 7 (a) is a corridor model diagram, Figure 7(b) is a sparse point cloud generated by traditional COLMAP, due to the lack of texture features in the corridor, the COLMAP reconstruction result only has one side of the corridor, and the other side of the corridor and the connecting part are abandoned, Figure 7 (c) is the reconstruction result of the present application, and the structure of the corridor is well restored by adding prior poses.

[0064] By comparison, it can be seen that the reconstruction result of the present application has significant improvement in structural integrity and detail restoration. Specifically, the point cloud generated by the traditional COLMAP may have problems of sparse point cloud and incomplete structure in some areas, especially in complex environments, and reconstruction deviation caused by insufficient feature matching is prone to occur. The optimized point cloud (since the prior information of the pose provided by the IMU is introduced, the point cloud density, structural integrity and geometric accuracy are obviously improved. In terms of accuracy, the point cloud generated by the traditional COLMAP may be unevenly distributed or deviated due to pose estimation error in some areas, while the point cloud constructed by the present application is more uniform in distribution and has higher geometric accuracy, which can more accurately restore the true structure of the three-dimensional scene. In addition, the point cloud constructed by the present application has significant improvement in reconstruction efficiency. The traditional COLMAP needs complex feature matching and geometric verification process, while the optimized method simplifies the calculation process by using known prior pose information, significantly shortening the reconstruction time.

[0065] Through this comparison, it can be clearly seen that the introduction of prior pose information optimizes the reconstruction of COLMAP sparse point cloud, especially in complex environment three-dimensional reconstruction. The optimized point cloud of the present application shows higher reconstruction efficiency and accuracy, provides higher quality input for subsequent 3DGS three-dimensional reconstruction, and further improves the overall reconstruction effect. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0067] Figure 1 is a flowchart of the improved method for Gaussian sputtering three-dimensional reconstruction based on prior pose provided by the embodiment of the present application;

[0068] Figure 2 is an algorithm flowchart of the present application;

[0069] Figure 3 is a prior pose estimation flowchart;

[0070] Figure 4 is a three-dimensional linear interpolation schematic diagram;

[0071] Figure 5 is a three-dimensional spherical interpolation schematic diagram;

[0072] Figure 6 is a sparse reconstruction flowchart of pose prior;

[0073] Figure 7 is a sparse reconstruction effect comparison diagram. DETAILED DESCRIPTION

[0074] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0075] Figure 1 is one of the process schematic diagrams of the improved method for Gaussian sputtering three-dimensional reconstruction based on pose prior provided by the embodiments of the present application.

[0076] As shown in Figure 1 , the improved method for Gaussian sputtering three-dimensional reconstruction based on pose prior provided by the embodiments of the present application mainly includes the following steps:

[0077] 11. The multi-modal information is collected by using the MarsLogger mobile phone application, and the multi-modal information includes video data and inertial measurement unit data;

[0078] 12. The pose information at the corresponding time point is estimated according to the inertial measurement unit data;

[0079] 13. The video data is frame-extracted and converted into pictures, and the pictures are accurately matched with the pose by using the three-dimensional linear interpolation and three-dimensional spherical interpolation methods according to the time stamp, so as to obtain the matched picture pose;

[0080] 14. The COLMAP sparse reconstruction based on pose prior is performed according to the matched picture pose, so as to obtain the sparse reconstruction point cloud;

[0081] 15. The 3DGS reconstruction is performed according to the sparse reconstruction point cloud.

[0082] In the embodiment of the present application, by introducing IMU data in the COLMAP sparse point cloud reconstruction process, the picture pose is estimated, and the estimated pose prior information is input into the COLMAP algorithm to optimize the sparse point cloud reconstruction result, and then the accuracy and integrity of the 3DGS three-dimensional reconstruction are improved; on the basis of the optimized sparse point cloud, the 3DGS algorithm is used for efficient and high-quality three-dimensional reconstruction, and finally the accurate three-dimensional model is obtained; this comprehensive method not only significantly improves the quality of sparse point cloud reconstruction, but also greatly improves the efficiency and reliability of 3DGS three-dimensional reconstruction, effectively solves the problem of poor reconstruction effect of the prior art in complex environment, in the traditional 3DGS reconstruction, the COLMAP feature extraction and reconstruction process is essential, but it is necessary to estimate the pose of the camera through complex feature matching and geometric verification, this process not only has large amount of calculation, but also is easy to be affected by the error accumulation of the pose estimation; and the present application, although it also needs to perform feature extraction and reconstruction process, does not need to perform complex rotation and translation calculation, because the known IMU provides the pose prior which can be directly used for reconstruction; specifically, after the camera pose is known, the pose does not need to be estimated by relying on complex feature matching and geometric verification process, which not only simplifies the calculation process, but also greatly speeds up the overall reconstruction speed; in terms of accuracy, due to the error accumulation problem of the pose estimation in the traditional method, the reconstruction accuracy is often reduced, and the present application uses the known accurate pose to effectively avoid this problem, and significantly improves the reconstruction accuracy; in addition, the prior pose is also optimized with image data, by taking the pose prior as a strong constraint condition, the optimization algorithm can converge to a more accurate solution faster, further improving the accuracy of the reconstruction result; at the same time, the present application also simplifies the reconstruction process, reduces the dependence on feature matching and initialization, reduces the risk of reconstruction failure caused by failure of pose estimation, makes the whole reconstruction process more stable and reliable, and provides a more effective solution for efficient and high-precision three-dimensional reconstruction; therefore, by combining the pose prior information estimation, COLMAP sparse point cloud reconstruction optimization and 3DGS three-dimensional reconstruction, the present application realizes efficient and accurate reconstruction of three-dimensional scene in complex environment such as mine tunnel, and provides strong technical support for research and application in related fields.

[0083] As shown in Figure 1 12, the pose information at the corresponding time point is estimated according to the inertial measurement unit data, including:

[0084] 121, obtaining and preprocessing the inertial measurement unit data to obtain a preprocessed inertial measurement unit data set, the inertial measurement unit data including a timestamp sequence, three-axis angular velocity of a gyroscope in an inertial measurement unit coordinate system and three-axis acceleration of an accelerometer;

[0085] 122, according to the inertial measurement unit data set, analyzing the acceleration data in the stationary state to detect the stationary state of the inertial measurement unit, to obtain the stationary state detection result;

[0086] 123, according to the stationary state detection result, analyzing the gravity acceleration measured by the accelerometer in the stationary state, and calculating the initial attitude of the inertial measurement unit;

[0087] 124, in the process of motion, using the gyroscope and accelerometer data to update the initial attitude of the inertial measurement unit, and setting the acceleration data K P to 0 in the stationary state to obtain the updated initial attitude of the inertial measurement unit; P

[0088] 125, according to the updated initial attitude of the inertial measurement unit, calculating the velocity and displacement relative to the initial position in the earth coordinate system, to obtain the reconstructed motion trajectory, that is, the pose information corresponding to the time point.

[0089] In the embodiment of the application, the IMU original measurement data set is obtained, and the data includes a time stamp sequence, three-axis angular velocity of the gyroscope in the IMU own coordinate system and three-axis acceleration of the accelerometer; in order to ensure data consistency, unit conversion is performed on the data, wherein the time is counted in seconds (s), the angular velocity is expressed in degrees per second (deg / s), and the acceleration is converted into gravity acceleration (g) as a unit; in addition, in order to reduce measurement noise and improve signal quality, low-pass filtering is applied to the acceleration data to smooth high-frequency noise, and high-pass filtering is applied to the angular velocity data to remove low-frequency drift; finally, based on actual requirements, appropriate data range is manually selected to ensure the accuracy and stability of the analysis;

[0090] ​Raw data is obtained from an inertial measurement unit (IMU), including a timestamp sequence, three-axis angular velocity of a gyroscope in an IMU coordinate system, and three-axis acceleration of an accelerometer, the raw data is preprocessed to ensure the accuracy and consistency of the data, the preprocessed inertial measurement unit dataset is obtained, providing a reliable basis for subsequent analysis and processing, the preprocessing step improves the accuracy of the data and reduces the accumulation of errors caused by noise or calibration errors, according to the preprocessed inertial measurement unit dataset, the characteristics of the acceleration data in the static state are analyzed, and whether the IMU is in the static state is judged by detecting the stability or specific mode of the acceleration data, the static state detection result is obtained, which provides a basis for subsequent attitude initialization and motion state judgment, and accurate detection of the static state helps to improve the accuracy and stability of attitude estimation, in the static state, the acceleration measured by the accelerometer mainly reflects the gravitational acceleration, the initial attitude of the IMU is calculated by analyzing the components of the gravitational acceleration in the IMU coordinate system, the initial attitude of the inertial measurement unit is obtained, which provides a starting reference for subsequent motion tracking, and accurate initial attitude estimation helps to reduce the cumulative error in the motion process, in the motion process, the attitude of the IMU is constantly updated using the data of the gyroscope and the accelerometer, in the static state, the acceleration data reflects the gravitational acceleration, so its gain is kept at a normal value, in the non-static state, to avoid the influence of the dynamic component in the accelerometer data on attitude estimation, the gain of the accelerometer data is set to 0, the updated initial attitude of the inertial measurement unit is obtained, which can reflect the motion state of the IMU in real time, the accuracy and robustness of attitude estimation are improved by reasonably processing the accelerometer data, the updated inertial measurement unit attitude is used, combined with the data of the gyroscope and the accelerometer, to calculate the velocity and displacement of the IMU in the earth coordinate system, the velocity is obtained by integrating the acceleration data, and the displacement is obtained by further integrating the velocity data, so as to reconstruct the motion trajectory of the IMU, the reconstructed motion trajectory, i.e. the pose information corresponding to the time point, is obtained, which provides accurate and continuous motion trajectory data for navigation, positioning, motion analysis and other applications.

[0091] As shown in Figure 1 122, according to the inertial measurement unit dataset, the acceleration data in the static state is analyzed to detect the static state of the inertial measurement unit, to obtain a static state detection result, including:

[0092] 1221, according to the inertial measurement unit dataset, the acceleration amplitude is calculated;

[0093] 1222, according to the acceleration amplitude, the cutoff frequency is set to remove the gravitational acceleration and static acceleration component, to obtain the high-pass filtered acceleration amplitude;

[0094] 1223, according to the high-pass filtered acceleration amplitude, taking the absolute value and low-pass filtering to obtain the low-pass filtered acceleration amplitude;

[0095] 1224, according to the low-pass filtered acceleration amplitude, threshold detection is performed, if the acceleration amplitude is less than the threshold, it is marked as a stationary state, if the acceleration amplitude is not less than the threshold, it is marked as a motion state, to obtain the stationary state detection result;

[0096] 1225, at the moment when the motion state is detected, the state change in the pre-set time range before and after is checked, and the switching between the stationary state and the motion state is smoothed by setting the margin value.

[0097] In the embodiment of the application, in the stationary state, the accelerometer of the IMU mainly measures the gravitational acceleration, and the angular velocity of the gyroscope should be close to 0; by analyzing the acceleration data in the stationary state, the initial attitude of the IMU can be calculated; in addition, during the motion process, the data in the stationary state can be used as a reference value for error detection and correction, thereby improving the accuracy and robustness of the pose estimation; the application detects whether the IMU is in a stationary state through the accelerometer, and the specific process is as follows: first, the acceleration amplitude is calculated, then the calculated acceleration amplitude is high-pass filtered, a very low cutoff frequency (0.001 Hz) is set to remove the gravitational acceleration and static acceleration components; then the absolute value is taken and low-pass filtered to remove the burst noise; finally, threshold detection is performed, if the amplitude is less than 0.03g, it is marked as a stationary state (1); otherwise, it is marked as a motion state (0); in order to further improve the accuracy and robustness of the stationary state detection, a front and rear edge smoothing process is adopted; specifically, at the moment when the motion state is detected, the state change in a certain time range before and after is checked, the switching between the stationary state and the motion state is smoothed by setting the margin value (margin), in the application, the margin value is set to 10% of the sampling rate, that is, 100 milliseconds; in this way, the misjudgment caused by instantaneous noise or motion change in a short time can be effectively reduced, thereby improving the stability of state recognition and ensuring the consistency of the stationary or motion state when the acceleration amplitude fluctuates slightly in a short time.

[0098] As shown in Figure 1 , 123, according to the stationary state detection result, the gravitational acceleration measured by the accelerometer in the stationary state is analyzed, and the initial attitude of the inertial measurement unit is calculated, including:

[0099] 1231, according to the stationary state detection result, assuming that the initial angle is 0, using Quaternion=q=[1,0,0,0] to represent the quaternion, wherein, Quaternion is representing the quaternion of the sensor relative to the earth, q is Quaternion representing the earth relative to the sensor, according to the conjugate relationship of the quaternion, the Quaternion=q * = [q 0’ -q 1’ -q 2’ -q3], wherein q0 is a real part, q1, q2, q3 are imaginary parts;

[0100] 1232, according to the static state detection result, the time window of the initial static state of the inertial measurement unit is selected, the average value of the measured acceleration vector in the static state in the window is calculated and normalized to obtain the unitized acceleration, wherein the measured acceleration in the static state is The unitized acceleration is a x , a y , a z is the acceleration in the x, y, z direction in the inertial measurement unit coordinate system; z

[0101] 1233, according to the quaternion and the unitized acceleration, when the carrier is static, using The relationship between the accelerometer output and the gravitational acceleration is calculated, and the estimated gravity direction is obtained using , wherein a x , a y , a z is the acceleration in the x, y, z direction in the inertial measurement unit coordinate system, q0 is a real part, q1, q2, q3 are imaginary parts, is the gravity direction, g is the gravitational acceleration, is the relationship between the accelerometer output and the gravitational acceleration;

[0102] 1234, according to the gravity direction of the unitized acceleration and the estimated gravity direction, the included angle error is calculated by cross product;

[0103] 1235, according to the included angle error, using InitError=K i ·InitError+K P ·Error correct the angular velocity of the gyroscope, and use to update the attitude of the inertial measurement unit, wherein IntError is the error integral term, K p and K i respectively represent the calculation error gain and integral error gain, Error is the error, is the updated quaternion, Δt is the time interval, t is the time, represents the derivative of the quaternion, which is calculated using quaternion multiplication to obtain the rate of change of the current attitude quaternion: represents the quaternion multiplication, s w t is the constructed angular velocity quaternion;

[0104] 1236, the above steps are iteratively performed until the error between the estimated gravity direction and the actual gravity direction is minimized to obtain an accurate initial attitude estimate, i.e., the initial attitude of the inertial measurement unit.

[0105] In the embodiment of the application, through the x and y coordinates of the point P(x, y, z), the algorithm can accurately calculate the position (gi, gj) in the grid, and further determine the grid index, so that the corresponding patch information of the column can be accurately obtained from the texture subsequently, providing a basis for the binary search; using the xyGridTexture texture, the algorithm can quickly obtain the patch information of the column, including the data position datapos, the patch quantity interfaceNum, and the in-order sequence index treeIndex, which provides necessary data support for the subsequent binary search, so that the search process can be efficiently performed; through twice binary search, the algorithm can accurately locate the position of the sampling point curPos in the patch sequence, the first binary search determines the patch range where the sampling point is located, and the second binary search finds the last patch different from the previous state, thereby determining the upper and lower relationship of the sampling point, and such accurate positioning enables the algorithm to accurately obtain the corresponding attribute value from the InterfaceUpAndDownProps array; the tree-shaped binary sampling method uses a texture to replace an array, fully utilizes the parallel processing capability of the GPU, improves the efficiency of data query, and accurate attribute value acquisition provides rich data support for volume rendering, so that the rendered image is more realistic and delicate; the tree-shaped binary sampling method improves the efficiency and effect of volume rendering through accurate positioning, efficient acquisition and accurate query, and provides strong support for the visualization processing of complex three-dimensional data.

[0106] As Figure 1 shown, 124, according to the updated initial attitude of the inertial measurement unit, the velocity and displacement relative to the initial position in the earth coordinate system are calculated to obtain the reconstructed motion trajectory, i.e., the pose information at the corresponding time point, including:

[0107] 1241, according to the updated initial attitude of the inertial measurement unit, the acceleration measured by the inertial measurement unit is converted into the earth reference system using to obtain the acceleration in the earth reference system, wherein, E a represents the acceleration in the earth reference system, S a represents the acceleration in the inertial measurement unit reference system, updating the quaternion, representing quaternion multiplication, representing the quaternion under the earth reference frame;

[0108] 1242, according to the acceleration under the earth reference frame, using v t = v t-1 + E a t ·Δt to calculate the velocity under the earth coordinate system, wherein, E a t represents the acceleration at time t, v t represents the velocity at time t, and Δt is the time interval;

[0109] 1243, according to the velocity under the earth coordinate system, combining the velocity change rate and the period of motion, using linear interpolation to calculate the velocity drift, and subtracting the velocity drift from the velocity under the earth coordinate system to obtain the corrected velocity;

[0110] 1244, according to the corrected velocity, initializing the position matrix to obtain the reconstructed motion trajectory, i.e. the pose information corresponding to the time point.

[0111] In the embodiment of the application, for the entire motion process, the quaternion is updated by using the gyroscope and accelerometer data. Since the acceleration data is more reliable in the stationary state, K p is assigned normally, and K p is set to 0 in the non-stationary state, and the rest of the updating process is the same as step 3. In addition, for the convenience of subsequent calculation, the conjugate of the quaternion is calculated and stored.

[0112] As shown in Figure 1 13, the video data is frame-extracted and converted into pictures, and the pictures are accurately matched with the poses by using three-dimensional linear interpolation and three-dimensional spherical interpolation methods according to the time stamps to obtain the matched picture poses, including:

[0113] 131, the video data is frame-extracted and converted into pictures to obtain a list of sampled video time stamps;

[0114] 132, according to the list of sampled video time stamps, the video frame time stamps not within the range of the inertial measurement unit time stamps are removed to obtain a list of valid video frame time stamps;

[0115] 133, according to the list of valid video frame time stamps, based on the positions and poses of the adjacent time inertial measurement units, the matched picture poses are obtained, wherein the position is realized by three-dimensional linear interpolation, and the pose is realized by quaternion interpolation, and the quaternion interpolation adopts spherical linear interpolation, and the formula is: wherein s is an interpolation factor, q0 is an inertial measurement unit attitude corresponding to a time stamp of an adjacent previous time, q1 is an inertial measurement unit attitude corresponding to a time stamp of an adjacent next time, and θ is an included angle between q0 and q1.

[0116] In the embodiment of the present application, the input video is converted into picture frames, the video sampling rate is about 30Hz, the mode of saving 1 frame every 6 frames is adopted, and 5 frames per second are approximately realized, so as to reduce the data amount, and a sampled video time stamp list is returned, which reflects the time flow of the video and provides basic data for subsequent time stamp screening and pose matching; the video frame time stamp is compared with the time stamp range of the inertial measurement unit (IMU), and the video frame time stamps not within the IMU time stamp range are removed, so as to obtain an effective video frame time stamp list synchronized with the IMU data in time, and ensure the consistency of the time effectiveness of the video frame and the IMU data in the subsequent pose matching process; according to the effective video frame time stamp list, the matched picture pose is obtained based on the position and attitude of the inertial measurement unit at adjacent times, the interpolation position corresponding to the video frame time stamp is calculated according to the position data of the IMU at adjacent times through three-dimensional linear interpolation, the interpolation attitude corresponding to the video frame time stamp is calculated according to the attitude data (represented by a quaternion) of the IMU at adjacent times through a spherical linear interpolation (Slerp) mode, the video frame and the IMU data are accurately matched in time and space, the corresponding position and attitude of each video picture are calculated, and a set of picture pose data corresponding one-to-one to the video frame time stamp is generated, which integrates the information of the video and the IMU, and provides accurate spatial and temporal positioning information for subsequent visual-inertial fusion, three-dimensional reconstruction, augmented reality and other applications.

[0117] As shown in Figure 1 , 14, COLMAP sparse reconstruction based on pose prior is performed according to the matched picture pose, and a sparse reconstruction point cloud is obtained, including:

[0118] 141, according to the matched picture pose, key points and descriptors are extracted from the input image through a detection algorithm, mutually corresponding feature point pairs are matched through the similarity of the descriptors, and geometric verification is performed, so as to obtain matched feature point pairs;

[0119] 142, according to the matched feature point pairs, all images are sorted, two images with the most matched feature point pairs are selected for initialization, and initialized images are obtained;

[0120] 143, according to the initialized images, the image with the most matched points or the widest distribution of visible points is selected, the pose of the new image is adjusted and optimized, and the image after incremental reconstruction is obtained;

[0121] 144, repeat the above steps, optimize the overall generated three-dimensional points and camera pose, and filter and remove the optimized point cloud and image, to obtain a sparse reconstruction point cloud.

[0122] In the embodiments of the present application, representative key points and descriptors are extracted from the images using detection algorithms, the matching of feature point pairs is performed through the similarity of the descriptors, and the accuracy of the matching is ensured through geometric verification, obtaining a set of verified and accurate matching feature point pairs, which provides a reliable basis for subsequent image sorting, initialization and incremental reconstruction; the images are sorted according to the number of matching feature point pairs, and the two images with the most matching points are selected for initialization, providing a stable starting point for subsequent incremental reconstruction, obtaining the initialized image pair, and the relative pose and feature point correspondence between the two images are determined, providing a reliable initial condition for the subsequent reconstruction process; on the basis of initialization, the image with the most matching points or the widest distribution of visible points is selected as the target for the next reconstruction, the pose of the new image is adjusted and optimized, and incremental three-dimensional reconstruction is realized, obtaining the incremental reconstructed images, which have been accurately aligned and fused in three-dimensional space with the previous images, forming a more complete three-dimensional scene; by repeating the above steps, the three-dimensional reconstruction scene is continuously expanded and improved, and finally the overall optimization of all generated three-dimensional points and camera poses is performed to improve the accuracy and consistency of the reconstruction, and at the same time, the filtered and removed point cloud and image are filtered and removed to remove noise and redundant information, obtaining a sparse reconstructed point cloud, which accurately represents the key structures and features in the three-dimensional scene, providing a reliable basis for subsequent dense reconstruction, texture mapping and other applications, and at the same time, through the filtering and removal process, the quality and usability of the point cloud are improved.

[0123] As shown in Figure 1 , 15, according to the sparse reconstruction point cloud, 3D GS reconstruction is performed, including:

[0124] 151, according to the sparse reconstruction point cloud, the point cloud and camera parameters are obtained by COLMAP sparse reconstruction, and the point cloud is initialized as a 3D Gaussian distribution, the 3D Gaussian distribution is G i ={μ i ,Σ i ,α i ,c i}, wherein μ i is the position, Σ i is the covariance matrix, α i is the opacity, and c i is the spherical harmonic coefficient;

[0125] 152, according to the 3D Gaussian distribution, the error between the reconstructed image and the training view is minimized using optimizing the parameters of the Gaussian distribution to obtain optimized 3D Gaussian distribution parameters, wherein, is an L1 loss, is a D-SSIM loss, and λ is a balance parameter;

[0126] 153. In the process of optimizing the 3D Gaussian distribution parameters, the number and density of the Gaussian distributions are dynamically adjusted through cloning and splitting operations;

[0127] 154. According to the optimized 3D Gaussian distribution parameters, real-time rendering is performed using a tile rasterization algorithm to obtain a rendered 3D Gaussian distribution;

[0128] 155. Repeat the steps of optimizing the 3D Gaussian distribution parameters and real-time rendering until a preset iteration number or convergence condition is reached.

[0129] In the embodiment of the present application, COLMAP tool is used for sparse reconstruction to obtain point cloud and camera parameters, providing basic data for subsequent 3D Gaussian distribution initialization, initializing point cloud as 3D Gaussian distribution, providing mathematical model for further optimization and rendering of point cloud, obtaining initial 3D Gaussian distribution parameters including position, covariance matrix, opacity and spherical harmonic coefficient, providing starting point for subsequent optimization and rendering process; by minimizing the error between the reconstructed image and the training view, the parameters of the 3D Gaussian distribution are optimized to make the reconstruction result closer to the real scene, obtaining optimized 3D Gaussian distribution parameters including more accurate position, covariance matrix, opacity and spherical harmonic coefficient, improving the accuracy and realism of the reconstructed point cloud; in the optimization process, the number and density of the Gaussian distributions are dynamically adjusted according to the complexity and detail requirements of the scene to adapt to the reconstruction requirements of different scenes, and the number and density of the Gaussian distributions are flexibly increased through cloning and splitting operations, improving the detail performance and scene adaptability of the reconstructed point cloud; using the tile rasterization algorithm, real-time rendering is performed according to the optimized 3D Gaussian distribution parameters to generate a realistic 3D scene, obtaining a rendered 3D Gaussian distribution, presenting a 3D scene with delicate details and realism, providing a basis for subsequent visual display and interaction; through continuous iteration of the optimization and rendering steps, the accuracy and realism of the reconstructed point cloud are gradually improved until a preset iteration number or convergence condition is reached, obtaining the final optimized 3D Gaussian distribution parameters and rendering result, and the accuracy and realism of the reconstructed point cloud are significantly improved, meeting the preset reconstruction requirements and standards.

[0130] As shown in Figure 1 153. In the process of optimizing the 3D Gaussian distribution parameters, the number and density of the Gaussian distributions are dynamically adjusted through cloning and splitting operations, including:

[0131] 1531, if the position gradient of small-scale Gaussian distribution is large, it indicates that the region needs more geometric details, and the number of Gaussian distributions is increased by cloning operation:

[0132] and α i <θ α , then clone

[0133] 1532, if the position gradient of large-scale Gaussian distribution is large, it indicates that the region needs finer division, and one large Gaussian distribution is split into two small Gaussian distributions by splitting operation:

[0134] and α i ≥θ α , then split

[0135] wherein μ i is position, α i is opacity, θ pos is the threshold of gradient, and θ α is the threshold of opacity.

[0136] In the embodiments of the present application, in the 3D Gaussian distribution parameter optimization process, the number and density of Gaussian distributions are dynamically adjusted by cloning and splitting operations. If the position gradient of small-scale Gaussian distribution is large, that is, the position change of the region is relatively sharp, it indicates that the region needs more geometric details to accurately describe. The number of Gaussian distributions is increased by cloning operation, that is, more similar Gaussian distributions are copied on the basis of the original small-scale Gaussian distribution to describe the geometric features of the region in more detail. The cloning operation increases the number of Gaussian distributions in the region that needs more geometric details, thereby improving the detail performance and accuracy of the reconstructed point cloud. This makes the reconstruction result closer to the real scene, especially in the region with complex geometric structure. If the position gradient of large-scale Gaussian distribution is large, that is, although the overall range of the region is large, the local change is also relatively sharp, it indicates that the region needs finer division to capture local features. One large Gaussian distribution is split into two small Gaussian distributions by splitting operation, that is, according to the direction of the position gradient, the original large-scale Gaussian distribution is split into two small Gaussian distributions with smaller scale and more specific position information. The splitting operation increases the density of Gaussian distributions in the region that needs finer division, thereby more accurately capturing local geometric features. This helps to improve the overall quality and realism of the reconstructed point cloud, especially in large-scale but locally complex scenes. By dynamically adjusting the number and density of Gaussian distributions through cloning and splitting operations, the number of Gaussian distributions can be flexibly increased or decreased according to the actual needs of the scene to improve the accuracy and realism of the reconstructed point cloud. This dynamic adjustment mechanism makes the reconstruction process more adaptable to the complexity and detail requirements of different scenes, thereby generating higher quality 3D reconstruction results.

[0137] As Figure 1 shown, 154, according to the optimized 3D Gaussian distribution parameters, real-time rendering is performed using a tile rasterization algorithm to obtain the rendered 3D Gaussian distribution, including:

[0138] 1541, according to the optimized 3D Gaussian distribution parameters, the screen is divided into 16x16 pixel tiles;

[0139] 1542, the Gaussian distribution in each tile is sorted using a GPU quicksort algorithm to obtain the sorted Gaussian distribution;

[0140] 1543, according to the sorted Gaussian distribution, rasterize each pixel and use to calculate the cumulative value of color and opacity, where p is the pixel position, C(p) is the color value of the pixel, N is the total number of samples, i is the index, μ i is the position, α i is the opacity, c i is the spherical harmonic coefficient, and T is the time.

[0141] In the embodiment of the application, according to the optimized 3D Gaussian distribution parameters, real-time rendering is performed using a tile rasterization algorithm to obtain the rendered 3D Gaussian distribution; according to the optimized 3D Gaussian distribution parameters, the screen is divided into 16x16 pixel tiles, which helps to divide the rendering task into smaller units for parallel processing and optimization, while reducing the number of Gaussian distributions within each tile and improving processing efficiency; the Gaussian distribution in each tile is sorted using a GPU quicksort algorithm to obtain the sorted Gaussian distribution, which ensures that the rendering can correctly handle the occlusion relationship between Gaussian distributions, so that the Gaussian distribution closer to the camera is rendered first, thereby generating correct visual effects and improving the accuracy and realism of the rendering result; according to the sorted Gaussian distribution, rasterize each pixel and use the formula to calculate the cumulative value of color and opacity, where p is the pixel position, C(p) is the color value of the pixel, N is the total number of samples, i is the index, μ i is the position, α i is the opacity, c i is the spherical harmonic coefficient, and T is the time; the rasterization process converts the Gaussian distribution into color and transparency information on the screen pixels, and through the accumulation of color and transparency, a rendering result with delicate details and realism can be generated, and the use of spherical harmonic coefficients can be used to handle complex visual effects such as lighting changes or surface reflections, while the time parameter can be used to achieve dynamic effects or animation.

[0142] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An improved method of Gaussian sputtering three-dimensional reconstruction based on pose prior, comprising: collecting multi-modal information using a MarsLogger mobile phone application, the multi-modal information including video data and inertial measurement unit data; estimating pose information at a corresponding time point according to the inertial measurement unit data; precisely matching the pictures with the poses by using three-dimensional linear interpolation and three-dimensional spherical interpolation methods according to timestamps to obtain matched picture poses, including: converting the video data into pictures by frame extraction to obtain a list of sampled video timestamps; according to the list of sampled video timestamps, eliminating video frame timestamps not within the range of inertial measurement unit timestamps to obtain a list of valid video frame timestamps; According to the valid video frame timestamp list, based on the position and posture of the adjacent time inertial measurement unit, the matched picture pose is obtained, wherein the position is achieved by three-dimensional linear interpolation, the posture is achieved by quaternion interpolation, the quaternion interpolation adopts spherical linear interpolation, and the formula is: Wherein s is an interpolation factor, is the inertial measurement unit posture corresponding to the adjacent previous time timestamp, is the inertial measurement unit posture corresponding to the adjacent next time timestamp, is and the included angle between them. performing COLMAP sparse reconstruction based on pose prior according to the matched picture poses to obtain a sparse reconstruction point cloud; performing 3DGS reconstruction according to the sparse reconstruction point cloud.

2. The method of claim 1, wherein the method further comprises: estimating pose information at a corresponding time point according to the inertial measurement unit data, including: obtaining inertial measurement unit data and preprocessing to obtain a set of preprocessed inertial measurement unit data, the inertial measurement unit data including a timestamp sequence, three-axis angular velocity of a gyroscope in an inertial measurement unit self-coordinate system, and three-axis acceleration of an accelerometer; analyzing acceleration data in a stationary state to detect the stationary state of the inertial measurement unit according to the set of inertial measurement unit data to obtain a stationary state detection result; calculating the initial attitude of the inertial measurement unit according to the acceleration measured by the accelerometer in the stationary state according to the stationary state detection result; During motion, gyroscope and accelerometer data are used to update the initial pose of the inertial measurement unit and in stationary state the acceleration data is set to 0 to get the updated pose of the inertial measurement unit is set to 0 to get the updated pose of the inertial measurement unit calculating the velocity and displacement in the earth coordinate system relative to the initial position according to the updated attitude of the inertial measurement unit to obtain the reconstructed motion trajectory, i.e. the pose information at the corresponding time point.

3. The method of claim 2, wherein the method further comprises: analyzing acceleration data in a stationary state to detect the stationary state of the inertial measurement unit according to the set of inertial measurement unit data to obtain a stationary state detection result, including: calculating the acceleration amplitude according to the set of inertial measurement unit data; setting a cutoff frequency to remove the gravity acceleration and static acceleration components according to the acceleration amplitude to obtain a high-pass filtered acceleration amplitude; taking the absolute value and performing low-pass filtering according to the high-pass filtered acceleration amplitude to obtain a low-pass filtered acceleration amplitude; performing threshold detection according to the low-pass filtered acceleration amplitude, if the acceleration amplitude is less than the threshold, it is marked as a stationary state, if the acceleration amplitude is not less than the threshold, it is marked as a motion state, to obtain a stationary state detection result; at the moment when the motion state is detected, check the state change in the preset time range before and after, and smooth the switching between the stationary state and the motion state by setting an edge value.

4. The method of claim 2, wherein the method further comprises: calculating the velocity and displacement in the earth coordinate system relative to the initial position according to the updated initial attitude of the inertial measurement unit to obtain the reconstructed motion trajectory, i.e. the pose information at the corresponding time point, including: According to the updated initial attitude of the inertial measurement unit, the acceleration measured by the inertial measurement unit is converted into the earth reference system to obtain the acceleration in the earth reference system, wherein, represents the acceleration in the earth reference system, represents the acceleration in the earth reference system, represents the acceleration in the earth reference system, is an updated quaternion, and represents quaternion multiplication, represents the quaternion in the earth reference system; The velocity in the earth coordinate system is calculated using the acceleration in the earth reference system, wherein a(t) denotes the acceleration at time t, v(t) denotes the velocity at time t, is the time interval; According to the velocity under the earth coordinate system, in combination with the rate of change of the velocity and the period of motion, the velocity drift is calculated by linear interpolation, and the velocity drift is subtracted from the velocity under the earth coordinate system to obtain the corrected velocity; According to the corrected velocity, the position matrix is initialized to obtain the reconstructed motion trajectory, that is, the pose information corresponding to the time point.

5. The method of claim 4, wherein the method further comprises: According to the sparse reconstructed point cloud, 3D GS reconstruction is carried out, including: According to the sparse reconstructed point cloud, a point cloud and camera parameters are obtained by sparse reconstruction using COLMAP, and the point cloud is initialized as a 3D Gaussian distribution, which is wherein, is a position, is a covariance matrix, is an opacity, is a spherical harmonic coefficient; According to the 3D Gaussian distribution, by minimizing the error between the reconstructed image and the training view, using optimizing the parameters of the Gaussian distribution to obtain the optimized 3D Gaussian distribution parameters, wherein, is an L1 loss, is a D-SSIM loss, is a balance parameter; In the 3D Gaussian distribution parameter optimization process, the number and density of Gaussian distribution are dynamically adjusted through cloning and splitting operations; According to the optimized 3D Gaussian distribution parameters, real-time rendering is carried out using the tile rasterization algorithm to obtain the rendered 3D Gaussian distribution; Repeat the above steps of 3D Gaussian distribution parameter optimization and real-time rendering until a preset iteration number or convergence condition is reached.

Images