Multi-view automatic driving scene reconstruction method based on three-dimensional Gaussian splashing

Abstract

The invention belongs to the technical field of intelligent transportation, and particularly relates to a multi-view automatic driving scene reconstruction method based on three-dimensional Gaussian splashing, which comprises the following steps: S1, data acquisition; s2, fast system initialization: setting the received first frame data as a key frame, and processing the first frame data by adopting a Gaussian high-pass filter to reserve most of high-frequency information in the image; s3, key frame selection, wherein dynamic key frame selection under a sliding window is adopted; s4, estimating the pose of the vehicle-mounted camera; and S5, reconstructing an automatic driving scene. According to the method, the precision and the efficiency of three-dimensional reconstruction are improved, the robustness and the real-time performance of the system are enhanced, high-fidelity rendering and dynamic scene editing are realized, and a comprehensive frequency domain analysis and regularization strategy is provided; and the method has a wide application prospect, can also be applied to the fields of high-precision map updating, environment perception and path planning, unmanned aerial vehicle navigation, robot navigation and the like, and provides powerful support for the development of intelligent transportation and automatic driving technologies in the future.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and in particular to a multi-view automatic driving scene reconstruction method based on three-dimensional Gaussian splashing. Background Art

[0002] With the rapid development of autonomous driving technology, the combination of vehicles and the cloud (vehicle-cloud integration) has become a trend. This combination can make full use of the powerful computing and storage capabilities of the cloud to improve the efficiency and reliability of the autonomous driving system. However, the current existing technologies still have some shortcomings in data collection, processing and transmission, which affects the real-time and accuracy of autonomous driving vehicles.

[0003] The invention patent application with the patent name "A multi-frame surround view autonomous driving scene simulation and editing method based on three-dimensional Gaussian" and the patent number "202311675493.4" discloses a method of three-dimensional reconstruction and simulation of large-scale autonomous driving scenes using multimodal sensor data. 3D Gaussian representation is used to represent the static background and the dynamic foreground with multiple objects respectively. It proposes to use incremental 3D static Gaussian fields to model large-scale long-time static backgrounds, and use 3D Gaussian dynamic graphs to represent dynamic multiple objects, and finally realize new view rendering through global 3D Gaussian splatting; at the same time, a controllable editing technology based on explicit 3D Gaussian is adopted to achieve efficient editing of large-scale autonomous driving scenes and generate realistic and diverse corner cases driving scenes.

[0004] The invention patent application with the patent name "A system and method for reconstructing three-dimensional scenes for intelligent driving based on binocular vision" and the patent number "202410032155.7" discloses a system and method for reconstructing three-dimensional scenes for intelligent driving. The camera is calibrated using Zhang Zhengyou's chessboard calibration method to obtain internal and external parameters of the camera; images are collected and the parameters obtained in the camera calibration process are used to perform distortion correction and stereo correction on the images; the Sobel operator is used to pre-process the image; the absolute value AD of the grayscale difference is used to calculate the matching cost; dynamic programming is used; the disparity map is processed through the reprojection matrix to obtain the three-dimensional coordinates of each point, and a three-dimensional point cloud map is displayed.

[0005] The invention patent with the name of "A method and system for constructing a digital twin scene for autonomous driving based on multi-view three-dimensional reconstruction" and the patent number of "202310123079.6" discloses a method for constructing a digital twin scene for autonomous driving, which collects and preprocesses multi-view images of the autonomous driving scene, and divides the processed image data into several groups; takes the collected multi-view images as input, outputs the position and posture corresponding to the camera that takes each image, thereby obtaining the internal and external parameter sequence of the camera; constructs a network model; uses the three processed feature points and the side lengths of the triangles they constitute to construct proportional triangular patches in the virtual three-dimensional space, and finds the positions of the three corresponding feature points in the multi-view three-dimensional reconstruction module, and simultaneously aligns the three feature points in the virtual point cloud model with the corresponding three points in the triangular patch, and scales the three-dimensional dense point cloud; divides the three-dimensional dense point cloud reconstructed from the images collected by the drone into an aerial point cloud model, and divides the three-dimensional dense point cloud reconstructed from other groups of images into a ground point cloud model, and aligns several ground point cloud models to the aerial point cloud model to form a final autonomous driving digital twin scene model.

[0006] However, the prior art has the following problems:

[0007] (1) Dependence on the accuracy of camera pose estimation: The accuracy of camera pose estimation directly affects the quality of 3D reconstruction. Once the camera pose estimation error is large, the subsequent reconstruction accuracy of depth map and point cloud will decrease.

[0008] (2) System initialization time is too long: System initialization includes multiple steps such as data acquisition, preprocessing, camera pose estimation, multi-view 3D reconstruction, point cloud model scale correction and air-to-ground point cloud model fusion. It takes a long time and seriously affects the real-time performance and response speed of the system.

[0009] (3) Dependence of scale correction of point cloud models: The scale correction of point cloud models depends on the accuracy of pre-selected feature points and triangular patches. Once the feature points are improperly selected or there is noise, the accuracy and consistency of the overall model will be affected. Summary of the invention

[0010] In order to overcome the defects of the above-mentioned prior art, the present invention proposes a multi-view autonomous driving scene reconstruction method based on three-dimensional Gaussian splashing, which not only improves the accuracy and efficiency of three-dimensional reconstruction, enhances the robustness and real-time performance of the system, but also realizes high-fidelity rendering and dynamic scene editing, and provides a comprehensive frequency domain analysis and regularization strategy. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0011] A multi-view autonomous driving scene reconstruction method based on three-dimensional Gaussian splashing includes the following steps:

[0012] S1 Data Collection: The collected data includes images of the vehicle's surroundings and timestamps. The collected data is transmitted to the vehicle's computing device via the vehicle's internal data bus or wireless communication;

[0013] S2 fast system initialization: The first frame of data received is set as the key frame, and the first frame of data is processed using a Gaussian high-pass filter to retain most of the high-frequency information in the image. First, the image I (x, y) is converted to the frequency domain representation F (u, v) through 2D discrete Fourier transform (2D DFT):

[0014]

[0015] Where (x, y) and (u, v) represent the coordinates in the image and its spectrum, respectively;

[0016] I(x, y) and F(u, v) represent the pixel value and the frequency domain representation of the image, respectively;

[0017] The frequency domain representation F(u, v) can be expressed as a function of amplitude |F(u, v)| and phase φ(F(u, v)):

[0018]

[0019]

[0020] Wherein, Real(F(u, v)) and Imag(F(u, v)) represent the real part and imaginary part of F(u, v), respectively;

[0021] Then pass through the high pass filter H high (u, c) effectively removes most of the low-frequency information in the image, retaining only the high-frequency components, generating a Gaussian low-pass filter H low (u, v), its expression is:

[0022]

[0023] Through the Gaussian low-pass filter, we can further obtain the Gaussian high-pass filter H high (u, c):

[0024]

[0025] Then the frequency domain image after high-pass filtering is obtained, and the spatial domain image I is obtained by inverse Fourier transform. high (x, y):

[0026] I high (x, y) = F -1 (F(I(x,y))·H high (u, v)); (6)

[0027] S3 keyframe selection: Dynamic keyframe selection under sliding window is adopted. For the current frame i and the last keyframe j, the covisibility is defined as the intersection over union (IoU) of the visible Gaussian model:

[0028]

[0029] Among them, G i and G j denote the Gaussian models visible in the i-th frame and the j-th frame respectively;

[0030] When IOU cov (i, j) is less than the set consensus threshold, the current frame i will be selected as the new key frame;

[0031] When the relative translation t between the current frame i and the last key frame j ij Exceeding the set relative displacement threshold kf m and the median depth of the i-th frame The product of , the current frame i will be selected as the new key frame:

[0032]

[0033] After a frame is determined to be a key frame, the frame needs to be added to the sliding window;

[0034] Maintain a sliding window of 10 units. If the window size exceeds 10 units, remove the least important keyframe based on the relative distance.

[0035] If the overlap coefficient between the current frame i and the latest keyframe is lower than the consensus threshold, the keyframe is deleted from the current window:

[0036]

[0037] Among them, G i and G j denote the Gaussian models visible in the i-th frame and the j-th frame respectively;

[0038] S4 Car-mounted Camera Pose Estimation: Car-mounted camera pose estimation aims to optimize the current camera pose. The camera pose is determined by adding a displacement to the previous pose. Then:

[0039] E t+1 =E t +(E t -E t-1 ); (10)

[0040] The initial pose estimation is completed in the first few seconds, and then the photometric residual and depth residual are minimized. The photometric residual measures the difference between the rendered image and the actual observed image, which is defined as follows:

[0041] E pho =||I(G,T CW )-I gt ||1; (11)

[0042] Among them, I(G, T CW ) represents the camera pose T CW Rendered Gaussian image G;

[0043] I gt is the actual observed image;

[0044] The depth residual refers to the difference between the rendered depth map and the actual observed depth map, which is defined as:

[0045]

[0046] Among them, D(G, T CW ) is obtained by the camera pose T CW The rendered depth map;

[0047] is the observed depth map;

[0048] By minimizing the residual, we can ensure that the difference between the rendered depth map and the observed depth map is minimized. In order to integrate the photometric residual and the geometric residual, the total loss function is:

[0049] E=λ pho E pho +λ dep E dep ; (13)

[0050] Among them, λ pho and λ dep is the equilibrium parameter;

[0051] At least 50 gradient descent iterations are performed for each frame, and the depth of each pixel is rasterized using alpha blending, with the following formula:

[0052]

[0053] Among them, z i is the average depth of Gaussian distribution i along the camera ray direction;

[0054] α i is the blending weight;

[0055] By using Lie algebra, we get the minimum Jacobian matrix, which can be obtained by the chain rule:

[0056]

[0057]

[0058] Among them, T CW Represents the 3D position of the Gaussian distribution in the camera coordinate system;

[0059] The partial derivative on the manifold is expressed as:

[0060]

[0061] in, Indicates group combination;

[0062] exp and log are the exponential and logarithmic mappings between Lie algebras and Lie groups, respectively;

[0063] Using the mapping we get:

[0064]

[0065] Where × represents the antisymmetric matrix of the 3D vector;

[0066] represents the i-th column of the matrix;

[0067] S5 autonomous driving scene reconstruction: physical points in 3D space The impact of is evaluated using the following function:

[0068]

[0069] Among them, o∈[0,1] is the opacity;

[0070] is the mean of the Gaussian distribution;

[0071] ∑ w =RSS T R T is the covariance matrix, which is represented by the scaling matrix and the rotation matrix calculate;

[0072] The 3D Gaussian is projected onto the 2D image plane for high-fidelity differentiable rendering, and the covariance matrix is ​​calculated by the following formula:

[0073] Σ′ w =JW∑ w W T J T ; (20)

[0074] Where W represents the sight direction;

[0075] J represents the projection transformation matrix;

[0076] The keyframe is used to reconstruct the current visible area. During the rendering process, a 3D Gaussian ellipsoid is used to project it onto a 2D image and calculate the color of each pixel:

[0077]

[0078] Among them, c i Indicates the color of the Gaussian ellipsoid;

[0079] C(p) is the sum of each c i The sum of , and represented by the function f i (p) weighting;

[0080] The depth rendering formula is as follows:

[0081]

[0082] Among them, d i represents the depth of the Gaussian ellipsoid;

[0083] Compare it with the input depth map and return the gradient of the 3D map;

[0084] Then render the contour image, the formula is as follows:

[0085]

[0086] Among them, f i (p) represents the influence of each Gaussian ellipsoid on pixel p;

[0087] The frequency regularization strategy is dynamically adjusted by using some changes in positioning and mapping indicators before and after rendering. Before rendering, dynamic frequency regularization is introduced. Each key frame is processed by low-pass filter and high-pass filter, and the frequency band range is dynamically adjusted according to ATE (Absolute Trajectory Error) and PSNR (Peak Signal-to-Noise Ratio). The coarse-to-fine Gaussian densification is gradually performed using the frequency components from low to high, and the frequency band range is dynamically adjusted. t The formula is as follows:

[0088]

[0089] Where D0 is the maximum range of the initial low-pass filter;

[0090] D is the minimum range of the high-pass filter;

[0091] i is the current progressive number;

[0092] A i-1 and P i-1 Represent the last progressive ATE normalization factor and PSNR normalization factor respectively;

[0093] When ATE is higher, D t The larger the value, the greater the D t Increasing the weight of high-frequency components can help capture more details and edge information, and help reduce posture errors;

[0094] The higher the PSNR, the better the t The smaller the D t The weight of low-frequency components will be increased, which can help preserve the overall structure of the image and reduce noise, thereby maintaining a high PSNR;

[0095] By adjusting D t , the frequency band range can be dynamically adjusted during training to balance low-frequency and high-frequency information; According to formula 1-5, we can get the Gaussian low-pass filter and high pass filter Then the filtered image I low (x, y) and I high (x, y) is:

[0096] I low (x, y) = F -1 (F(I(x,y))·H low (u, v)); (25)

[0097] I high (x, y) = F -1 (F(I(x,y))·H high (u, v)); (26)

[0098] The low-pass and high-pass filtered images are combined to obtain the reconstructed image I updated (x, y), and then the reconstructed image I updated (x, y) is rendered and the frequency regularization loss L is calculated fre :

[0099] I updated (x, y) = I low (x,y)+I high (x, y); (27)

[0100] L fre =|| low -I gt||1+||I high -I gt ||1; (28)

[0101] Among them, I gt is a real image;

[0102] During the frequency regularization process after rendering, the PSNR, SSIM, and LPIPS values ​​of the image are first calculated, and these values ​​are used to decide whether to perform regularization:

[0103] S=w PSNR P psnr +w SSIM S ssim +w LPIPS L lpips ; (29)

[0104] Among them, w PSNR , w SSIM and w LPIPS They are the weights of PSNR, SSIM and LPIPS respectively;

[0105] P psnr 、s ssim and L lpips is the corresponding normalization factor;

[0106] The frequency band range is dynamically adjusted by combining the three evaluation indicators of PSNR, SSIM and LPIPS. The frequency amplitude and phase difference between the rendered image and the true value image are obtained according to formula 1-6, which is set as the low-frequency amplitude difference D la , low frequency phase difference D lp , high frequency amplitude difference D ha and high frequency phase difference D hp ;

[0107]

[0108]

[0109]

[0110]

[0111] Among them, LF(u, v) and HF(u, v) represent the low-frequency and high-frequency components of the rendered image, respectively;

[0112] and Respectively represent the low-frequency and high-frequency components of the true value image;

[0113] Assume the current frequency range is:

[0114]

[0115] Spectral regularization loss l fre It is expressed as a weighted sum of the differences in low-frequency and high-frequency amplitudes and phases:

[0116] l fre =ω1(D la +D lp )+ω2(D ha +D hp ); (35)

[0117] Among them, ω1 and ω2 are the weights of the low-frequency and high-frequency parts respectively;

[0118] In order to avoid the artifact problem caused by the elongation of the Gaussian distribution in the line of sight, isotropic regularization is introduced:

[0119]

[0120] Among them, s i represents the scaling parameter of the i-th Gaussian;

[0121] is its mean;

[0122] Suppose the optimization objective function including RGB, depth image loss, isotropic regularization and regularization loss is:

[0123] L=λ pho L pho +λ dep L dep +λ iso L iso +λ Fre L fre +λ fre l fre ; (37)

[0124] Among them, L pho =||I(G,T CW )-I gt ||1 is the photometric loss, which measures the L1 difference between the rendered color image and the true color image at the current camera pose;

[0125] is the geometric depth loss, which is the L1 difference between the rendered depth map and the optimized prior depth map;

[0126] λ pho , dep , iso and λ Fre and fre are the weight factors of each loss item respectively;

[0127] Based on historical data, dynamic progressive frequency regularization from low-frequency to high-frequency information is achieved. As the training process proceeds, there is a learning process that gradually shifts from image contours to image edges and details, thereby achieving Gaussian densification mapping from coarse to fine.

[0128] Furthermore, in step S3, the consensus threshold is set at the beginning of the program, and its value range is 0.3-0.9.

[0129] Furthermore, in step S3, the relative displacement threshold kf is set m It is set at the beginning of the program and its value range is 0-0.5.

[0130] The present invention not only improves the accuracy and efficiency of 3D reconstruction, enhances the robustness and real-time performance of the system, but also realizes high-fidelity rendering and dynamic scene editing, and provides comprehensive frequency domain analysis and regularization strategies; it has broad application prospects and can also be applied to other intelligent driving application fields such as high-precision map updating, environmental perception and path planning, drone navigation, robot navigation, etc., providing strong support for the future development of intelligent transportation and autonomous driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0132] Figure 1 It is a flow chart of the present invention, such as Figure 1 As shown, a multi-view autonomous driving scene reconstruction method based on three-dimensional Gaussian splashing includes the following steps:

[0133] S1 Data Collection: On the vehicle side, there are various image data collection devices, including multi-camera systems and depth cameras installed in smart vehicles. The collected data includes images of the vehicle's surroundings and timestamps. The collected data is transmitted to the vehicle's computing device via the vehicle's internal data bus or wireless communication.

[0134] S2 Fast System Initialization: Initial map construction, set the first frame of data received as the key frame, and use Gaussian high-pass filter to process the first frame of data to retain most of the high-frequency information in the image. The high-frequency information contains the details and edge features in the image. These features play a key role in the subsequent calculation of photometric errors. First, convert the image I (x, y) to the frequency domain representation F (u, v), which is achieved through 2D discrete Fourier transform (2D DFT):

[0135]

[0136] Where (x, y) and (u, v) represent the coordinates in the image and its spectrum, respectively;

[0137] I(x, y) and F(u, v) represent the pixel value and the frequency domain representation of the image, respectively;

[0138] The frequency domain representation F(u, v) can be expressed as a function of amplitude |F(u, v)| and phase φ(F(u, v)):

[0139]

[0140]

[0141] Among them, Real(F(u, v)) and Imag(F(u, c)) represent the real part and imaginary part of F(u, v) respectively;

[0142] Then pass through the high pass filter H high (u, v) effectively removes most of the low-frequency information in the image and only retains the high-frequency components. On this basis, a Gaussian low-pass filter H is generated. low (u, v),

[0143] Through the Gaussian low-pass filter, we can further obtain the Gaussian high-pass filter H high (u, v):

[0144]

[0145] Then the frequency domain image after high-pass filtering is obtained, and the spatial domain image I is obtained by inverse Fourier transform. high (x, y):

[0146] I high (x, y) = F -1 (F(I(x,y))·H high (u, v)); (6)

[0147] This approach reduces the computational effort of pose estimation and map construction during initialization, while preserving high-frequency features that are critical for photometric error computation.

[0148] S3 key frame selection: Dynamic key frame selection is adopted under the sliding window. The key frame selection is mainly based on the field of view coverage and relative displacement conditions between frames. When added to the sliding window, the window size needs to be dynamically managed to avoid redundant key frames.

[0149] For the current frame i and the last key frame j, the covisibility is defined as the Intersection over Union (IoU) of the visible Gaussian model:

[0150]

[0151] Among them, G i and G j denote the Gaussian models visible in the i-th frame and the j-th frame respectively;

[0152] When IOU cov (i, j) is less than the set consensus threshold, then the current frame i will be selected as the new key frame. The consensus threshold is set at the beginning of the program, and its value range is 0.3-0.9. This threshold is used to ensure that the newly selected key frame has enough new information in the field of view;

[0153] When the relative translation t between the current frame i and the last key frame j ij Exceeding the set relative displacement threshold kf m and the median depth of the i-th frame The product of the relative displacement threshold kf is set m It is set at the beginning of the program, and its value range is 0-0.5, then the current frame i will be selected as the new key frame:

[0154]

[0155] After a frame is determined to be a key frame, the frame needs to be added to the sliding window;

[0156] Maintain a sliding window of 10 units. If the window size exceeds 10 units, remove the least important keyframe based on the relative distance to ensure that the window size is within a reasonable range.

[0157] If the overlap coefficient between the current frame i and the latest keyframe is lower than the consensus threshold, the keyframe is deleted from the current window:

[0158]

[0159] Among them, G i and G j denote the Gaussian models visible in the i-th frame and the j-th frame, respectively.

[0160] S4 Car-mounted Camera Pose Estimation: Car-mounted camera pose estimation aims to optimize the current camera pose. The camera pose is determined by adding a displacement to the previous pose. Then:

[0161] E t+1 =E t+(E t -E t-1 ); (10)

[0162] The initial pose estimation is completed in the first few seconds, and then the difference between the rendered image and the observed image is minimized and the error between the rendered depth map and the actual observed depth map is minimized by minimizing the photometric residual and the depth residual. The photometric residual measures the difference between the rendered image and the actual observed image, which is defined as follows:

[0163] E pho =||I(G,T CW )-I gt ||1; ( 11)

[0164] Among them, I(G, T CW ) represents the camera pose T CW Rendered Gaussian image G;

[0165] I gt is the actual observed image;

[0166] The camera pose can then be further optimized by minimizing the depth residual, which is the difference between the rendered depth map and the actual observed depth map, and is defined as:

[0167]

[0168] Among them, D(G, T CW ) is obtained by the camera pose T CW The rendered depth map;

[0169] D is the observed depth map;

[0170] By minimizing the residual, we can ensure that the difference between the rendered depth map and the observed depth map is minimized. In order to integrate the photometric residual and the geometric residual, the total loss function is:

[0171] E=λ pho E pho +λ dep E dep ; (13)

[0172] Among them, λ pho and λ dep It is a balance parameter used to weigh the weight of photometric residual and geometric residual in the total loss;

[0173] To achieve accurate tracking, at least 50 gradient descent iterations are performed for each frame, and the depth of each pixel is rasterized via alpha blending, with the following formula:

[0174]

[0175] Where zi is the average depth of Gaussian distribution i along the camera ray direction;

[0176] α, is the mixing weight;

[0177] Through this method, the depth information can be effectively refined and the depth observation accuracy of the system can be improved.

[0178] We derive the first explicit Jacobian matrix for the SE(3) camera pose of the 3D Gaussian distribution used in EWA splatting and 3DGS. By using Lie algebra, we obtain the minimum Jacobian matrix, ensuring that its dimensions match the degrees of freedom and avoiding redundant calculations. By using the chain rule, we obtain:

[0179]

[0180]

[0181] Among them, T CW Represents the 3D position of the Gaussian distribution in the camera coordinate system;

[0182] The partial derivative on the manifold is expressed as:

[0183]

[0184] in, Indicates group combination;

[0185] exp and log are the exponential and logarithmic mappings between Lie algebras and Lie groups, respectively;

[0186] Using these mappings we get:

[0187]

[0188] Where × represents the antisymmetric matrix of the 3D vector;

[0189] Represents the i-th column of the matrix.

[0190] S5 Autonomous Driving Scene Reconstruction: The present invention uses 3D Gaussian Splatting as the representation method of the map of the present invention. The representation function of Gaussian distribution for a single 3D Gaussian in 3D space is as follows:

[0191] For physical points in 3D space The impact of is evaluated using the following function:

[0192]

[0193] Among them, o∈[0,1] is the opacity;

[0194] is the mean of the Gaussian distribution;

[0195] ∑ w =RSS T R T is the covariance matrix, which is represented by the scaling matrix and the rotation matrix calculate;

[0196] The 3D Gaussian is projected onto the 2D image plane for high-fidelity differentiable rendering, allowing for Gaussian scene optimization and camera pose estimation. The covariance matrix is ​​calculated using the following formula:

[0197] ∑' w =JWΣ w W T J T ; (20)

[0198] Where W represents the sight direction;

[0199] J represents the projection transformation matrix;

[0200] The mapping process aims to maintain the consistency of the 3D structure and optimize the newly inserted Gaussian distribution. The keyframe is used to reconstruct the current visible area. During the rendering process, the 3D Gaussian ellipsoid is used to project it onto the 2D image and calculate the color of each pixel:

[0201]

[0202] Among them, c i Indicates the color of the Gaussian ellipsoid;

[0203] C(p) is the sum of each c i The sum of , and represented by the function f i (p) weighting;

[0204] The depth rendering formula is as follows:

[0205]

[0206] Among them, d i represents the depth of the Gaussian ellipsoid;

[0207] Compare it with the input depth map and return the gradient of the 3D map;

[0208] To determine visibility, a silhouette image is also rendered, with the following formula:

[0209]

[0210] Among them, f i (p) represents the influence of each Gaussian ellipsoid on pixel p, which is calculated in a similar way to the influence function of the 3D Gaussian distribution mentioned above.

[0211] The present invention proposes a coarse-to-fine layered map construction strategy, which uses the dynamic frequency regularization strategy based on the changes of some indicators of positioning and mapping before and after rendering. Before rendering, dynamic frequency regularization is introduced, and each key frame is processed by a low-pass filter and a high-pass filter. The frequency band range is dynamically adjusted according to ATE (Absolute Trajjectory Error) and PSNR (Peak Signal-to-Noise Ratio), and Gaussian densification from coarse to fine is gradually performed using frequency components from low to high. The purpose of dynamic frequency regularization is to perform high-pass and low-pass filtering on the image through frequency domain analysis to enhance the details and edges of the image while maintaining the integrity of the overall structure, and dynamically adjust the frequency band range. t The formula is as follows:

[0212]

[0213] Where D0 is the maximum range of the initial low-pass filter;

[0214] D is the minimum range of the high-pass filter;

[0215] i is the current progressive number;

[0216] A i-1 and P i-1 Represent the last progressive ATE normalization factor and PSNR normalization factor respectively;

[0217] When ATE is higher, D t The larger the value, the greater the D t Increasing the weight of high-frequency components can help capture more details and edge information, and help reduce posture errors;

[0218] The higher the PSNR, the better the t The smaller the D t The weight of low-frequency components will be increased, which can help preserve the overall structure of the image and reduce noise, thereby maintaining a high PSNR;

[0219] By adjusting D t , the frequency band range can be dynamically adjusted during training to balance low-frequency and high-frequency information; According to formula 1-5, we can get the Gaussian low-pass filter and high pass filtering Device Then the filtered image I low (x, y) and I high (x, y) is:

[0220] I low (x, y) = F -1 (F(I(x,y))·H low (u, v)); (25)

[0221] I high (x, y) = F -1 (F(I(x,y))·H high (u, v)); (26)

[0222] The low-pass and high-pass filtered images are combined to obtain the reconstructed image I updated (x, y), and then the reconstructed image I updated (x, y) is rendered and the frequency regularization loss L is calculated fre :

[0223] I updated (x, y) = I low (x,y)+I high (x, y); (27)

[0224] L fre =||I low -I gt ||1+||I high -I gt ||1; (28)

[0225] Among them, I gt is a real image;

[0226] During the frequency regularization process after rendering, the PSNR, SSIM, and LPIPS values ​​of the image are first calculated, and these values ​​are used to decide whether to perform regularization:

[0227] S=w PSNR P psnr +w SSIM S ssim +w LPIPS L lpips ; (29)

[0228] Among them, w PSNR , w SSIM and w LPIPS are the weights of PSNR, SSIM and LPIPS respectively; P psnr 、s ssim and L lpips is the corresponding normalization factor.

[0229] The present invention further optimizes the frequency regularization strategy, especially in the post-rendering stage, by combining the three evaluation indicators of PSNR, SSIM and LPIPS to dynamically adjust the frequency band range. This optimization aims to better balance the overall structure of the image and the retention of detail information, and reduce the artifacts and noise introduced in the rendering process; the frequency amplitude and phase difference between the rendered image and the true value image are obtained according to formula 1-6, which is set as the low-frequency amplitude difference Dl a , low frequency phase difference Dl p , high frequency amplitude difference D ha and high frequency phase difference D hp ;

[0230]

[0231]

[0232]

[0233]

[0234] Among them, LF(u, v) and HF(u, v) represent the low-frequency and high-frequency components of the rendered image, respectively;

[0235] and Respectively represent the low-frequency and high-frequency components of the true value image;

[0236] In order to dynamically adjust the frequency band range, set the current frequency band range to:

[0237]

[0238] Spectral regularization loss l fre It is expressed as a weighted sum of the differences in low-frequency and high-frequency amplitudes and phases:

[0239] l fre =ω1(D la +D lp )+ω2(D ha +D hp ); (35)

[0240] Among them, ω1 and ω2 are the weights of the low-frequency and high-frequency parts respectively;

[0241] In order to avoid the artifact problem caused by the elongation of the Gaussian distribution in the line of sight, isotropic regularization is introduced:

[0242]

[0243] Among them, s i represents the scaling parameter of the i-th Gaussian;

[0244] is its mean;

[0245] Suppose the optimization objective function including RGB, depth image loss, isotropic regularization and regularization loss is:

[0246] L=λ pho L pho +λ dep L dep +λ iso L iso +λ Fre L fre +λ fre l fre ; (37)

[0247] Among them, L pho =||I(G,T CW )-I gt ||1 is the photometric loss, which measures the L1 difference between the rendered color image and the true color image at the current camera pose;

[0248] is the geometric depth loss, which is the L1 difference between the rendered depth map and the optimized prior depth map;

[0249] λ pho , dep , iso and λ Fre and fre are the weight factors of each loss item respectively;

[0250] The algorithm of the present invention realizes dynamic progressive frequency regularization from low-frequency to high-frequency information according to historical data. As the training process proceeds, the algorithm of the present invention has a learning process that gradually shifts from image contours to image edges and details, thereby realizing Gaussian densification mapping from coarse to fine.

[0251] The present invention has the following technical effects:

[0252] (1) Improved accuracy and efficiency of 3D reconstruction: By using Gaussian high-pass filters and Gaussian low-pass filters to process image data, low-frequency noise is effectively removed and high-frequency details are retained, thereby improving the accuracy of 3D reconstruction. At the same time, the dynamic frequency regularization strategy enables the system to adaptively adjust the frequency bandwidth during initialization and subsequent processing, further improving the reconstruction efficiency.

[0253] (2) Enhanced robustness and real-time performance of the system: The dynamic keyframe selection strategy under the sliding window and the on-board camera pose estimation method can quickly respond to environmental changes and improve the robustness and real-time performance of the system. The camera pose estimation process combines the minimization of photometric residual and depth residual, so that the system can still maintain high accuracy under different lighting conditions.

[0254] (3) High-fidelity rendering and dynamic scene editing: Through 3D Gaussian splashing technology, 3D Gaussians are projected onto the 2D image plane to achieve high-fidelity differentiable rendering, which allows for Gaussian scene optimization and camera pose estimation. Based on the controllable editing technology of explicit 3D Gaussians, efficient editing of large-scale autonomous driving scenes is achieved, and realistic and diverse corner case driving scenes are generated, thereby assisting in the training and verification of autonomous driving systems, improving the performance of autonomous driving systems in extreme scenarios and ensuring safety.

[0255] (4) It provides a comprehensive frequency domain analysis and regularization strategy: by dynamically adjusting the frequency band range and integrating the three evaluation indicators of PSNR, SSIM and LPIPS, the frequency domain image is regularized to reduce the artifacts and noise introduced in the rendering process and maintain the overall structure and detail information of the image.

[0256] The present invention has broad application prospects in the field of autonomous driving, and can be further applied to the following aspects:

[0257] (1) High-precision map update: By continuously updating vehicle trajectory data, high-precision maps are updated in real time to support the navigation and decision-making of the autonomous driving system, thereby improving the navigation accuracy and driving safety of autonomous vehicles in complex road conditions.

[0258] (2) Environmental perception and path planning: Based on the three-dimensional reconstruction and dynamic scene reconstruction capabilities of the present invention, the environmental perception capability and path planning accuracy of the autonomous driving vehicle are improved, achieving a more intelligent and safe autonomous driving experience.

[0259] (3) The technical method of the present invention can also be extended to other intelligent driving application fields such as drone navigation and robot navigation to enhance their navigation and operation capabilities in complex environments.

[0260] Through the above expansion scheme, the present invention can not only improve the performance of the autonomous driving system, but also play an important role in a wider range of intelligent transportation and intelligent driving applications, providing strong support for the future development of intelligent transportation and autonomous driving technologies.

[0261] In summary, the present invention not only improves the accuracy and efficiency of three-dimensional reconstruction, enhances the robustness and real-time performance of the system, but also realizes high-fidelity rendering and dynamic scene editing, and provides comprehensive frequency domain analysis and regularization strategies; it has broad application prospects and can also be applied to other intelligent driving application fields such as high-precision map updates, environmental perception and path planning, drone navigation, robot navigation, etc., providing strong support for the future development of intelligent transportation and autonomous driving technology.

Claims

1. A multi-view autonomous driving scene reconstruction method based on three-dimensional Gaussian splashing, characterized in that: The following steps are involved: S1 Data Collection: The collected data includes images of the vehicle's surroundings and timestamps. The collected data is transmitted to the vehicle's computing device via the vehicle's internal data bus or wireless communication; S2 fast system initialization: The first frame of data received is set as the key frame, and the first frame of data is processed using a Gaussian high-pass filter to retain most of the high-frequency information in the image. First, the image I (x, y) is converted to the frequency domain representation F (u, v) through 2D discrete Fourier transform (2D DFT): Where (x, y) and (u, v) represent the coordinates in the image and its spectrum, respectively; I(x, y) and F(u, v) represent the pixel value and the frequency domain representation of the image, respectively; The frequency domain representation F(u, v) can be expressed as a function of amplitude |F(u, v)| and phase φ(F(u, v)): Wherein, Real(F(u, v)) and Imag(F(u, v)) represent the real part and imaginary part of F(u, v), respectively; Then pass through the high pass filter H high (u, v) effectively removes most of the low-frequency information in the image, retaining only the high-frequency components, generating a Gaussian low-pass filter H low (u, v), its expression is: Through the Gaussian low-pass filter, we can further obtain the Gaussian high-pass filter H high (u, v): Then the frequency domain image after high-pass filtering is obtained, and the spatial domain image I is obtained by inverse Fourier transform. high (x, y): I high (x,y)=F -1 (F(I(x,y))·H high (u,v)); (6) S3 keyframe selection: Dynamic keyframe selection under sliding window is adopted. For the current frame i and the last keyframe j, the covisibility is defined as the intersection over union (IoU) of the visible Gaussian model: Among them, G i and G j denote the Gaussian models visible in the i-th frame and the j-th frame respectively; When IOU cov (i, j) is less than the set consensus threshold, the current frame i will be selected as the new key frame; When the relative translation t between the current frame i and the last key frame j ij Exceeding the set relative displacement threshold kf m and the median depth of the i-th frame The product of , the current frame i will be selected as the new key frame: After a frame is determined to be a key frame, the frame needs to be added to the sliding window; Maintain a sliding window of 10 units. If the window size exceeds 10 units, remove the least important keyframe based on the relative distance. If the overlap coefficient between the current frame i and the latest keyframe is lower than the consensus threshold, the keyframe is deleted from the current window: Among them, G i and G j denote the Gaussian models visible in the i-th frame and the j-th frame respectively; S4 Car-mounted Camera Pose Estimation: Car-mounted camera pose estimation aims to optimize the current camera pose. The camera pose is determined by adding a displacement to the previous pose. Then: AND t+1 =And t +(And t -AND t-1 ); (10) The initial pose estimation is completed in the first few seconds, and then the photometric residual and depth residual are minimized. The photometric residual measures the difference between the rendered image and the actual observed image, which is defined as follows: E pho =‖I(G,T CW )-I gt ||1; (11) Among them, I(G, T CW ) represents the camera pose T CW Rendered Gaussian image G; Igt is the actual observed image; The depth residual refers to the difference between the rendered depth map and the actual observed depth map, which is defined as: Among them, D(G, T CW ) is obtained by the camera pose T CW The rendered depth map; is the observed depth map; By minimizing the residual, we can ensure that the difference between the rendered depth map and the observed depth map is minimized. In order to integrate the photometric residual and the geometric residual, the total loss function is: And=λ pho AND pho +λ dep AND dep ; (13) Among them, λ pho and λ dep is the equilibrium parameter; At least 50 gradient descent iterations are performed for each frame, and the depth of each pixel is rasterized using alpha blending, with the following formula: Among them, z i is the average depth of Gaussian distribution i along the camera ray direction; α i is the blending weight; By using Lie algebra, we get the minimum Jacobian matrix, which can be obtained by the chain rule: Among them, T CW Represents the 3D position of the Gaussian distribution in the camera coordinate system; The partial derivative on the manifold is expressed as: in, Indicates group combination; exp and log are the exponential and logarithmic mappings between Lie algebras and Lie groups, respectively; Using the mapping we get: Where × represents the antisymmetric matrix of the 3D vector; represents the i-th column of the matrix; S5 autonomous driving scene reconstruction: physical points in 3D space The impact of is evaluated using the following function: Among them, o∈[0,1] is the opacity; is the mean of the Gaussian distribution; ∑ w =RSS T R T is the covariance matrix, which is represented by the scaling matrix and the rotation matrix calculate; The 3D Gaussian is projected onto the 2D image plane for high-fidelity differentiable rendering, and the covariance matrix is ​​calculated by the following formula: ∑′ w =JW∑ w W T J T ; (20) Where W represents the sight direction; J represents the projection transformation matrix; The keyframe is used to reconstruct the current visible area. During the rendering process, a 3D Gaussian ellipsoid is used to project it onto a 2D image and calculate the color of each pixel: Among them, c i Indicates the color of the Gaussian ellipsoid; C(p) is the sum of each c i The sum of , and represented by the function f i (p) weighting; The depth rendering formula is as follows: Among them, d i represents the depth of the Gaussian ellipsoid; Compare it with the input depth map and return the gradient of the 3D map; Then render the contour image, the formula is as follows: Among them, f i (P) represents the influence of each Gaussian ellipsoid on pixel p; The frequency regularization strategy is dynamically adjusted by using some changes in positioning and mapping indicators before and after rendering. Before rendering, dynamic frequency regularization is introduced. Each key frame is processed by low-pass filter and high-pass filter, and the frequency band range is dynamically adjusted according to ATE (Absolute Trajectory Error) and PSNR (Peak Signal-to-Noise Ratio). The coarse-to-fine Gaussian densification is gradually performed using the frequency components from low to high, and the frequency band range is dynamically adjusted. t The formula is as follows: Where D0 is the maximum range of the initial low-pass filter; D is the minimum range of the high-pass filter; i is the current progressive number; A i-1 and P i-1 Represent the last progressive ATE normalization factor and PSNR normalization factor respectively; When ATE is higher, D t The larger the value, the greater the D t Increasing the weight of high-frequency components can help capture more details and edge information, and help reduce posture errors; The higher the PSNR, the better the t The smaller the D t The weight of low-frequency components will be increased, which can help preserve the overall structure of the image and reduce noise, thereby maintaining a high PSNR; By adjusting D t , the frequency band range can be dynamically adjusted during training to balance low-frequency and high-frequency information; According to formula 1-5, we can get the Gaussian low-pass filter and high pass filter Then the filtered image I low (x, y) and I high (x, y) is: I low (x,y)=F -1 (F(I(x,y))·H low (u,v)); (25) I high (x,y)=F -1 (F(I(x,y))·H high (u,v)); (26) The low-pass and high-pass filtered images are combined to obtain the reconstructed image I updated (x, y), and then the reconstructed image I updated (x, y) is rendered and the frequency regularization loss L is calculated fre : I updated (x,y)=I low (x,y)+I high (x,y); (27) L fre =‖I low -I gt ||1+||I high -I gt ||1; (28) Among them, I gt is a real image; During the frequency regularization process after rendering, the PSNR, SSIM, and LPIPS values ​​of the image are first calculated, and these values ​​are used to decide whether to perform regularization: S=w psNR P psnr +w SSIM S ssim +w LPIPS L lpips ; (29) Among them, w PSNR , w SSIM and w LPIPS They are the weights of PSNR, SSIM and LPIPS respectively; P psnr 、s ssim and L lpips is the corresponding normalization factor; The frequency band range is dynamically adjusted by combining the three evaluation indicators of PSNR, SSIM and LPIPS. The frequency amplitude and phase difference between the rendered image and the true value image are obtained according to formula 1-6, which is set as the low-frequency amplitude difference D la , low frequency phase difference D lp , high frequency amplitude difference D ha and high frequency phase difference D hp ; Among them, LF(u, v) and HF(u, v) represent the low-frequency and high-frequency components of the rendered image, respectively; and Respectively represent the low-frequency and high-frequency components of the true value image; Assume the current frequency range is: Spectral regularization loss l fre It is expressed as a weighted sum of the differences in low-frequency and high-frequency amplitudes and phases: L fre =ω1(D la +D lp )+ω2(D ha +D hp ); (35) Among them, ω1 and ω2 are the weights of the low-frequency and high-frequency parts respectively; In order to avoid the artifact problem caused by the elongation of the Gaussian distribution in the line of sight, isotropic regularization is introduced: Among them, s i represents the scaling parameter of the i-th Gaussian; is its mean; Suppose the optimization objective function including RGB, depth image loss, isotropic regularization and regularization loss is: L=λ pho L pho +λ dep L dep +λ iso L iso +λ Fre L fre +λ fre l fre (37) Among them, L pho =||I(G,T CW )-I gt ||1 is the photometric loss, which measures the L1 difference between the rendered color image and the true color image at the current camera pose; is the geometric depth loss, which is the L1 difference between the rendered depth map and the optimized prior depth map; λ pho , dep , iso and λ Fre and fre are the weight factors of each loss item respectively; Based on historical data, dynamic progressive frequency regularization from low-frequency to high-frequency information is achieved. As the training process proceeds, there is a learning process that gradually shifts from image contours to image edges and details, thereby achieving Gaussian densification mapping from coarse to fine.

2. The multi-view autonomous driving scene reconstruction method based on three-dimensional Gaussian splashing according to claim 1 is characterized in that: In step S3, the consensus threshold is set at the beginning of the program, and its value range is 0.3-0.

9.

3. The multi-view autonomous driving scene reconstruction method based on three-dimensional Gaussian splashing according to claim 1 is characterized in that: In step S3, the relative displacement threshold kf is set m It is set at the beginning of the program and its value range is 0-0.5.

Images