Three-dimensional gaussian city scene reconstruction method and system based on point cloud depth diffusion

Abstract

The application discloses a three-dimensional Gaussian city scene reconstruction method and system based on point cloud depth diffusion. The method comprises the following steps: obtaining a target scene with a labeled data set for reconstructing a scene model; processing laser radar point cloud data in the data set using a depth diffusion method to obtain dense point cloud and dense depth map; selecting an appropriate number of dense point cloud as the initialization position of the three-dimensional Gaussian using a voxel selection method; when training the three-dimensional Gaussian model, using the dense depth map for depth supervision, additionally adding a dense depth loss function and a loss coefficient acting on the overall loss function; and obtaining the trained explicit city scene three-dimensional Gaussian model. The application fully considers the characteristics of the three-dimensional Gaussian city scene reconstruction task, and is suitable for the characteristics, proposes a more suitable initialization method and depth supervision method, and can obtain better scene modeling effect.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction, and in particular to a 3D Gaussian scene reconstruction method based on point cloud depth diffusion. Background Technology

[0002] Recent advancements in end-to-end autonomous driving have highlighted the importance of closed-loop evaluation. However, existing simulators inevitably exhibit domain differences from the real world, emphasizing the need for real-world closed-loop simulators and driving the development of high-quality urban scene modeling methods. Neural Radiation Field (NeRF)-based methods and 3D Gaussian Splashing (3DGS)-based methods are leading technologies in this field, providing realistic rendering for new perspectives. In particular, 3DGS-based methods, through explicit modeling and efficient differentiable splashing, have boosted rendering speed to real-time levels and are currently being widely adopted.

[0003] LiDAR point clouds, with their accurate depth priors, are widely used in 3DGS-based autonomous driving methods, primarily serving two purposes: Gaussian initialization, as 3DGS heavily relies on initialized point clouds. Compared to structure-of-motion (SfM) points extracted from motion, LiDAR points are denser, helping to reduce textureless areas and thus promoting the densification process; and depth supervision, as depth is crucial for 3DGS scene modeling because it directly determines the Gaussian's 3D spatial position. LiDAR depth provides effective constraints during training, reducing artifacts and improving rendering quality. However, LiDAR point clouds remain too sparse, lacking sufficient surface contours and geometry during initialization. Furthermore, statistics show that, on average, only 0.68% of image pixels receive depth supervision. Especially in scenes with image distortion, such as under exposure or low-light conditions, relying solely on color and sparse depth supervision leads to a significant performance degradation. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method for reconstructing three-dimensional Gaussian urban scenes based on point cloud depth diffusion.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] In a first aspect, the present invention provides a method for reconstructing a three-dimensional Gaussian city scene based on point cloud depth diffusion, which includes the following steps:

[0007] S1. Obtain a training dataset for the target scene used to reconstruct the scene model. The training dataset contains a spatiotemporally continuous image sequence of the target scene, corresponding camera parameters, and LiDAR point cloud and SfM point cloud data.

[0008] S2. Project the sparse LiDAR point cloud onto the image space of each image in the image sequence. Traverse the image space and treat each pixel as a target pixel. Select the depths of the nearest multiple point clouds to each target pixel. Calculate the weights based on the distance between each point cloud and the target pixel. Then, sum the depths of the selected multiple point clouds according to the calculated weights to obtain the depth value of the target pixel. After traversing all pixels in the image space, obtain the dense depth map corresponding to the image. Project the dense depth map onto the world coordinate system using the camera pose parameters to obtain the depth point cloud.

[0009] S3. Use the voxel selection method to uniformly sample the depth point cloud, and combine the sampled point cloud with the SfM point cloud and the lidar point cloud in the dataset to form a dense point cloud and add it to the training dataset as the initial position of the 3D Gaussian, so as to balance the modeling speed and modeling quality.

[0010] S4. Train a 3D Gaussian model using the training dataset. During the training process, add an additional dense depth loss function to the depth rendering result. The final loss is the weighted sum of image reconstruction loss, SSIM loss and dense depth loss multiplied by the loss coefficient. Input the dense depth map and perform supervised optimization on the 3D Gaussian model based on the final loss to improve the modeling quality of the 3D Gaussian model for the scene.

[0011] S5. Visualize the trained 3D Gaussian model to obtain explicit modeling results.

[0012] Preferably, the training dataset in step S1 includes multiple temporally and spatially continuous image samples collected from the target scene, and each image sample contains a corresponding camera intrinsic parameter matrix. Camera external parameters SfM point cloud P SfM and LiDAR point cloud P LiDAR ,in It is a rotation matrix. It is a translation vector.

[0013] Preferably, the specific steps for obtaining the dense depth map and depth point cloud for each image in the image sequence in step S2 are as follows:

[0014] S21, convert the sparse lidar point cloud P containing L points into a single point cloud. LiDAR The point set is obtained by projecting from the world coordinate system to the image coordinate system. in This represents the pixel coordinates in the image coordinate system obtained by projecting the i-th point. This represents the depth corresponding to the i-th point after projection;

[0015] S22. For each pixel p = (u, v, d) in the current image of the image sequence, use the top-k algorithm to extract the pixel from the point set. Select the k nearest LiDAR depth points to pixel p in the image coordinate system. At the same time, it is necessary to ensure that for any selected lidar depth point and any unselected lidar depth point satisfy:

[0016]

[0017] Where ||·||2 represents the Euclidean distance.

[0018] Represents Λ p,k for The complement;

[0019] S23. For each LiDAR depth point at a different distance from pixel p Calculate the corresponding weights using the following formula.

[0020]

[0021] Where τ is a constant;

[0022] For all weight Normalization is performed to obtain each Final weight Used for k lidar depth points depth Perform a weighted summation to obtain a dense depth map. The diffusion depth D(u,v) of each pixel p = (u,v,d):

[0023]

[0024] S24. Dense depth map By projecting the camera intrinsic matrix K and camera extrinsic parameters [R,T] back to the world coordinate system, the depth point cloud P corresponding to the current image is obtained. depth .

[0025] Preferably, the constant τ = 1 in step S23.

[0026] Preferably, when uniformly sampling the depth point cloud using the voxel selection method in step S3, the three-dimensional space of the depth point cloud is divided into a voxel grid according to a preset voxel size, and one depth point at the center of each voxel replaces all the depth point cloud inside the voxel, thereby obtaining a uniformly sampled point cloud.

[0027] Preferably, when uniformly sampling the depth point cloud using a voxel selection method, the depth point cloud P is controlled by adjusting the voxel size. depth The density of midpoint cloud sampling.

[0028] Preferably, the calculation steps for the final loss in step S4 are as follows:

[0029] S41. Project the 3D Gaussian model from each training iteration onto the camera coordinate system using the camera extrinsic parameters [R,T] to obtain the rendering depth for the current iteration. Where H represents the pixel height of the image, and W represents the pixel width of the image;

[0030] S42. Calculate the dense depth loss of the image using the following formula:

[0031]

[0032] Where (u,v) are the image pixel coordinates, and ||1 represents the L1 norm;

[0033] S43. Record the dense depth loss for each image in the training dataset to obtain the dense depth loss vector l∈R. M Where M represents the number of images in the training set; the loss coefficient ξ∈R is obtained by normalizing the dense depth loss vector l. M :

[0034] ξ = γ × Sgmoid(l × β)

[0035] Where γ and β are two control constants, and Sgmoid(·) represents the Sigmoid normalization operation;

[0036] S44. Calculate the image reconstruction loss for the i-th image in the training dataset. SSIM loss Then, it is compared with the dense depth loss in step S42. Together, they constitute the overall loss. Simultaneously, the i-th element ξ(i) of the loss coefficient ξ is applied to the overall loss to obtain the final loss for the i-th image in the training dataset:

[0037]

[0038] Where λ1 and λ2 are adjustable weight hyperparameters.

[0039] Preferably, the weight values ​​γ = 2 and β = 100 in step S43.

[0040] Preferably, the weight values ​​λ1 = 0.2 and λ2 = 0.01 in step S44.

[0041] Preferably, in step S5, the trained 3D Gaussian model is visualized using rasterization technology.

[0042] Secondly, the present invention provides a three-dimensional Gaussian urban scene reconstruction system based on point cloud depth diffusion, which includes the following modules:

[0043] The data acquisition module is used to acquire a training dataset of the target scene for reconstructing the scene model. The training dataset contains a spatiotemporally continuous image sequence of the target scene, corresponding camera parameters, and LiDAR point cloud and SfM point cloud data.

[0044] The depth point cloud generation module projects sparse LiDAR point clouds onto the image space of each image in the image sequence. It traverses the image space, treating each pixel as a target pixel, and selects the depths of multiple nearest neighbor point clouds for each target pixel. It calculates weights based on the distance between each point cloud and the target pixel, and then sums the depths of the selected point clouds according to the calculated weights to obtain the depth value of the target pixel. After traversing all pixels in the image space, it obtains the dense depth map corresponding to the image. The dense depth map is then projected onto the world coordinate system using camera pose parameters to obtain the depth point cloud.

[0045] The initialization position generation module is used to uniformly sample the depth point cloud using a voxel selection method, and to combine the sampled point cloud with the SfM point cloud and lidar point cloud in the dataset to form a dense point cloud and add it to the training dataset as the initial position of the 3D Gaussian, so as to balance modeling speed and modeling quality.

[0046] The model training module is used to train a 3D Gaussian model using the training dataset. During the training process, an additional dense depth loss function is added to the depth rendering result. The final loss is the weighted sum of image reconstruction loss, SSIM loss and dense depth loss multiplied by the loss coefficient. The dense depth map is input and the 3D Gaussian model is supervised and optimized based on the final loss to improve the modeling quality of the 3D Gaussian model for the scene.

[0047] The visualization module is used to visualize the trained 3D Gaussian model and obtain explicit modeling results.

[0048] Thirdly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0049] The memory is used to store computer programs;

[0050] The processor is configured to, when executing the computer program, implement the three-dimensional Gaussian city scene reconstruction method based on point cloud depth diffusion as described in any of the first aspects above.

[0051] This invention proposes a depth initialization and supervision scheme more suitable for 3D scene reconstruction tasks. It employs a depth diffusion method to densify sparse LiDAR point clouds, resulting in improved modeling capabilities for 3D scene reconstruction models. Compared to traditional 3D reconstruction schemes, this invention offers the following advantages:

[0052] First, this invention proposes a feasible method for reconstructing 3D urban scenes based on point cloud depth diffusion.

[0053] Secondly, this invention fully considers the characteristics of 3D reconstruction tasks and specifically designs the lidar point cloud depth diffusion in point cloud initialization and depth map supervision, and designs the corresponding dense depth loss function and overall depth loss coefficient.

[0054] Finally, the three-dimensional urban scene reconstruction method proposed in this invention has wide applicability. As a plug-in method, it can be easily extended to existing general methods and achieve good results. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the three-dimensional urban scene reconstruction method based on point cloud depth diffusion according to the present invention.

[0056] Figure 2 This is a schematic diagram of the module composition of the three-dimensional urban scene reconstruction system based on point cloud depth diffusion according to the present invention.

[0057] Figure 3 This is a schematic diagram of the composition of the computer electronic device in this invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0059] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0060] This invention discloses a 3D urban scene reconstruction method based on point cloud depth diffusion, which assists in the initialization and depth supervision of a 3D Gaussian model by densifying sparse LiDAR point clouds. It is important to note that the specific type of the 3D Gaussian model in this invention is not limited, and while urban scene reconstruction is a preferred task, it is not actually limited to urban scene reconstruction. The structure of the 3D Gaussian model in this invention is allowed to vary. (Reference) Figure 1 In a preferred embodiment of the present invention, the 3D scene reconstruction method based on point cloud depth diffusion specifically includes the following steps:

[0061] S1. Obtain a training dataset for the target scene used to reconstruct the scene model. The training dataset contains a spatiotemporally continuous image sequence of the target scene, corresponding camera parameters, and LiDAR point cloud and SfM point cloud data.

[0062] In step S1 of this embodiment, the training dataset includes multiple temporally and spatially continuous image samples collected from the target scene, and each image sample should be accompanied by a corresponding camera intrinsic parameter matrix. Camera external parameters SfM point cloud P SfM and LiDAR point cloud P LiDAR ,in It is a rotation matrix. It is a translation vector. It should be noted that the multiple image samples that are continuous in time and space in this invention are essentially equivalent to videos continuously captured by an imaging device while moving within the target scene. Image frames can be extracted from the videos to form the aforementioned spatiotemporally continuous image sequence in the training dataset.

[0063] Furthermore, in embodiments of the present invention, to test the new perspective generation effect of scene reconstruction according to the present invention, a corresponding test dataset can be set up based on the above training dataset. Generally, the test dataset is constructed by extracting frames from the training dataset. In the present invention, the final algorithm goal is to render the completed 3D model into a color image from the actual training set camera perspective or the test set camera perspective, respectively demonstrating its scene reconstruction capability and new perspective generation capability.

[0064] S2. Project the sparse LiDAR point cloud onto the image space of each image in the image sequence. Traverse the image space and treat each pixel as a target pixel. Select the depths of the nearest multiple point clouds to each target pixel. Calculate the weights based on the distance between each point cloud and the target pixel. Then, sum the depths of the selected multiple point clouds according to the calculated weights to obtain the depth value of the target pixel. After traversing all pixels in the image space, obtain the dense depth map corresponding to the image. Project the dense depth map onto the world coordinate system using the camera pose parameters to obtain the depth point cloud.

[0065] In step S2 of this embodiment, the acquisition of dense depth map and depth point cloud mainly includes the following sub-steps:

[0066] The specific steps for obtaining dense depth maps and depth point clouds for each image in an image sequence are as follows:

[0067] S21, convert the sparse lidar point cloud P containing L points into a single point cloud. LiDAR The point set is obtained by projecting from the world coordinate system to the image coordinate system. in This represents the pixel coordinates in the image coordinate system obtained by projecting the i-th point. This represents the depth corresponding to the i-th point after projection.

[0068] S22. For each pixel p = (u, v, d) in the current image of the image sequence, use the top-k algorithm to extract the pixel from the point set. Select the k nearest LiDAR depth points to pixel p in the image coordinate system. At the same time, it is necessary to ensure that for any selected lidar depth point and any unselected lidar depth point satisfy:

[0069]

[0070] Where ||·||2 represents the Euclidean distance.

[0071] Represents Λ p,k for The complement;

[0072] S23. For each LiDAR depth point at a different distance from pixel p Calculate the corresponding weights using the following formula.

[0073]

[0074] in It is what is sought. The corresponding weights; σ represents the distance between two pixels; 2 It is the variance of the Gaussian function that controls The diffusion range; τ is a constant. In the embodiments of the present invention, τ = 1.

[0075] For all weight Normalization is performed to obtain each Final weight Used for k lidar depth points depth Perform a weighted summation to obtain a dense depth map. The diffusion depth D(u,v) of each pixel p = (u,v,d):

[0076]

[0077] S24. Dense depth map By projecting the camera intrinsic matrix K and camera extrinsic parameters [R,T] back to the world coordinate system, the depth point cloud P corresponding to the current image is obtained. depth .

[0078] S3. Use the voxel selection method to uniformly sample the depth point cloud, and combine the sampled point cloud with the SfM point cloud and the lidar point cloud in the dataset to form a dense point cloud and add it to the training dataset as the initial position of the 3D Gaussian, so as to balance the modeling speed and modeling quality.

[0079] In step S3 of this embodiment, when uniformly sampling the depth point cloud using the voxel selection method, the specific method is to divide the three-dimensional space of the depth point cloud into a voxel grid according to a preset voxel size, and replace all the depth point cloud inside the voxel with one depth point at the center of each voxel, thereby obtaining a uniformly sampled point cloud. Since only one point is retained for each voxel, the depth point cloud P is controlled by adjusting the voxel size. depth The density of midpoint cloud sampling. In an embodiment of the present invention, the voxel size is 0.1m × 0.1m × 0.2m when dividing the voxel grid.

[0080] S4. Train a 3D Gaussian model using the training dataset. During the training process, add a dense depth loss function and a loss coefficient that acts on the overall loss to the depth rendering result. The final loss is the weighted sum of the image reconstruction loss, SSIM loss and dense depth loss multiplied by the loss coefficient. Input the dense depth map and perform supervised optimization on the 3D Gaussian model based on the final loss to improve the modeling quality of the 3D Gaussian model for the scene.

[0081] It should be noted that the aforementioned 3D Gaussian model is existing technology. In the embodiments of this invention, it can be implemented using GaussianPro. Its input is a spatiotemporally continuous image sequence of the target scene, and a dense point cloud composed of sampled point clouds, SfM point clouds, and LiDAR point clouds (used to initialize the 3D Gaussian model). The aforementioned sampled point clouds, SfM point clouds, and LiDAR point clouds can be directly superimposed to form a dense point cloud.

[0082] In step S4 of this embodiment, the calculation steps for the dense depth loss function, the overall loss, and the final loss are as follows:

[0083] S41. Project the 3D Gaussian model from each training iteration onto the camera coordinate system using the camera extrinsic parameters [R,T] to obtain the rendering depth for the current iteration. Where H represents the pixel height of the image, and W represents the pixel width of the image;

[0084] S42. Calculate the dense depth loss of the image using the following formula:

[0085]

[0086] Where (u,v) are the image pixel coordinates, and ||1 represents the L1 norm;

[0087] S43. Record the dense depth loss for each image in the training dataset to obtain the dense depth loss vector l∈R. M Where M represents the number of images in the training set; the loss coefficient ξ∈R is obtained by normalizing the dense depth loss vector l. M :

[0088] ξ = γ × Sgmoid(l × β)

[0089] Where γ and β are two control constants, and Sgmoid(·) represents the Sigmoid normalization operation. In the embodiments of the present invention, the weight values ​​γ = 2 and β = 100 are set.

[0090] S44. Calculate the image reconstruction loss for the i-th image in the training dataset. SSIM loss Then, it is compared with the dense depth loss in step S42. Together, they constitute the overall loss. Simultaneously, the i-th element ξ(i) of the loss coefficient ξ is applied to the overall loss to obtain the final loss for the i-th image in the training dataset:

[0091]

[0092] Where λ1 and λ2 are adjustable weight hyperparameters. In the embodiments of the present invention, the weight values ​​are set to λ1 = 0.2 and λ2 = 0.01.

[0093] It should be noted that this invention can actually be combined with other 3D reconstruction methods as a plug-in approach. When combined with other general methods, the loss function specific to the corresponding method can also be added to the final loss function form. and the corresponding weight λ new .

[0094] S5. Visualize the trained 3D Gaussian model to obtain explicit modeling results.

[0095] It should be noted that visualizing a three-dimensional Gaussian model is an existing technology, and in the embodiments of this invention, it can be achieved using techniques such as rasterization.

[0096] It should be noted that the method steps S1 to S5 described above can essentially be implemented in the form of a computer program.

[0097] Therefore, based on the same inventive concept, this invention also provides a 3D Gaussian city scene reconstruction system based on point cloud depth diffusion, corresponding to the point cloud depth diffusion-based 3D Gaussian city scene reconstruction method provided in the above embodiments, such as... Figure 2 As shown, the system includes:

[0098] The data acquisition module is used to acquire a training dataset of the target scene for reconstructing the scene model. The training dataset contains a spatiotemporally continuous image sequence of the target scene, corresponding camera parameters, and LiDAR point cloud and SfM point cloud data.

[0099] The depth point cloud generation module projects sparse LiDAR point clouds onto the image space of each image in the image sequence. It traverses the image space, treating each pixel as a target pixel, and selects the depths of multiple nearest neighbor point clouds for each target pixel. It calculates weights based on the distance between each point cloud and the target pixel, and then sums the depths of the selected point clouds according to the calculated weights to obtain the depth value of the target pixel. After traversing all pixels in the image space, it obtains the dense depth map corresponding to the image. The dense depth map is then projected onto the world coordinate system using camera pose parameters to obtain the depth point cloud.

[0100] The initialization position generation module is used to uniformly sample the depth point cloud using a voxel selection method, and to combine the sampled point cloud with the SfM point cloud and lidar point cloud in the dataset to form a dense point cloud and add it to the training dataset as the initial position of the 3D Gaussian, so as to balance modeling speed and modeling quality.

[0101] The model training module is used to train a 3D Gaussian model using the training dataset. During the training process, an additional dense depth loss function is added to the depth rendering result. The final loss is the weighted sum of image reconstruction loss, SSIM loss and dense depth loss multiplied by the loss coefficient. The dense depth map is input and the 3D Gaussian model is supervised and optimized based on the final loss to improve the modeling quality of the 3D Gaussian model for the scene.

[0102] The visualization module is used to visualize the trained 3D Gaussian model and obtain explicit modeling results.

[0103] In the three-dimensional Gaussian city scene reconstruction system based on point cloud depth diffusion in the above embodiment, the specific process executed by each module can also refer to the specific steps of S1 to S5 mentioned above, and will not be repeated here.

[0104] Similarly, based on the same inventive concept, this invention also provides a computer electronic device corresponding to the three-dimensional Gaussian city scene reconstruction method based on point cloud depth diffusion provided in the above embodiments, such as... Figure 3 As shown, it includes a memory and a processor;

[0105] The memory is used to store computer programs;

[0106] The processor is configured to implement the three-dimensional Gaussian city scene reconstruction method based on point cloud depth diffusion as described above when executing the computer program.

[0107] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0108] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0109] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0110] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0111] The following section applies the three-dimensional Gaussian city scene reconstruction method based on point cloud depth diffusion, as shown in S1 to S5 above, to a specific example to demonstrate its technical effects.

[0112] Example

[0113] The implementation method of this embodiment is as described in S1 to S5 above, and the specific steps will not be elaborated in detail again. The following only demonstrates its effect based on example data. This invention is implemented on a dataset with lidar point cloud annotations:

[0114] The Waymo Open dataset contains approximately 12 million LiDAR bounding boxes and approximately 12 million image bounding boxes, generating approximately 113k LiDAR object trajectories and approximately 250,000 image trajectories. The dataset is divided into 1000 training sets and 150 test sets.

[0115] This example uses the Waymo Open dataset to test 11 challenging scenarios, and the validation metrics chosen are FPS, PSNR, SSIM, and LPIPS. FPS (Frames Per Second) is a metric for measuring the smoothness of video, animation, or game visuals. In this example, it's used as a rendering speed metric when rendering a 3D model into a 2D scene image; a higher FPS value indicates faster rendering. PSNR (Peak Signal-to-Noise Ratio) is an objective metric for measuring image quality. It measures image quality by comparing the mean squared error (MSE) between the original and distorted images. A higher PSNR value indicates less distortion and higher image quality. SSIM (Structural Similarity Index Measure) is a metric for measuring the structural similarity between two images. SSIM considers the human visual system's perception of structural information, better reflecting the details and texture of an image. SSIM values ​​range from 0 to 1; the closer the value is to 1, the better the image quality. The advantage of SSIM is that it considers brightness, contrast, and structure, aligning with human visual perception. The disadvantages are that the computation is relatively complex and requires a large amount of computation. Also, for images with minor distortion, the SSIM metric may not provide accurate evaluation results. LPIPS (Learned Perceptual Image Patch Similarity) is a deep learning-based image quality assessment method. LPIPS evaluates image quality by training a deep neural network to simulate human visual perception of images. The core idea of ​​the LPIPS algorithm is based on perceptual learning; that is, by training a deep neural network, it can simulate and learn the features of human visual perception and evaluate image quality based on these features. The advantages of LPIPS are that it can well simulate human visual perception of images and is relatively sensitive to distortions in details and textures. At the same time, this method has good generalization ability and can evaluate image quality in different scenes.

[0116] In this embodiment, the existing 3D Gaussian Pro model is selected as the base model, and the method of the present invention is applied to this base model (hereinafter referred to as the method of the present invention). The test results of the two in the scene reconstruction task are shown in the table below:

[0117] Table 1: Average test results of scene reconstruction in 11 test scenarios of the Waymo Open dataset

[0118]

[0119] The above experiments demonstrate that the method of this invention can significantly improve the model reconstruction performance of existing general methods, while only causing a slight decrease in rendering speed, which is negligible in practical situations. It can be well implemented for 3D reconstruction applications in urban scenes and has significant application value.

[0120] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for reconstructing a 3D Gaussian city scene based on point cloud depth diffusion, characterized in that, Includes the following steps: S1. Obtain a training dataset for the target scene used to reconstruct the scene model. The training dataset contains a spatiotemporally continuous image sequence of the target scene, corresponding camera parameters, and LiDAR point cloud and SfM point cloud data. S2. Project the sparse LiDAR point cloud onto the image space of each image in the image sequence. Traverse the image space and treat each pixel as a target pixel. Select the depths of the nearest multiple point clouds to each target pixel. Calculate the weights based on the distance between each point cloud and the target pixel. Then, sum the depths of the selected multiple point clouds according to the calculated weights to obtain the depth value of the target pixel. After traversing all pixels in the image space, obtain the dense depth map corresponding to the image. Project the dense depth map onto the world coordinate system using the camera pose parameters to obtain the depth point cloud. S3. Use the voxel selection method to uniformly sample the depth point cloud, and combine the sampled point cloud with the SfM point cloud and the lidar point cloud in the dataset to form a dense point cloud and add it to the training dataset as the initial position of the 3D Gaussian, so as to balance the modeling speed and modeling quality. S4. Train a 3D Gaussian model using the training dataset. During the training process, add an additional dense depth loss function to the depth rendering result. The final loss is the weighted sum of image reconstruction loss, SSIM loss and dense depth loss multiplied by the loss coefficient. Input the dense depth map and perform supervised optimization on the 3D Gaussian model based on the final loss to improve the modeling quality of the 3D Gaussian model for the scene. S5. Visualize the trained 3D Gaussian model to obtain explicit modeling results; In step S2, the specific steps for obtaining the dense depth map and depth point cloud for each image in the image sequence are as follows: S21, will include Sparse lidar point cloud of individual points The point set is obtained by projecting from the world coordinate system to the image coordinate system. ,in This represents the pixel coordinates in the image coordinate system obtained by projecting the i-th point. This represents the depth corresponding to the i-th point after projection. S22. For each pixel in the current image of the image sequence. Using the top-k algorithm from the point set Select the nearest pixel in the image coordinate system k lidar depth points At the same time, it is necessary to ensure that for any selected lidar depth point and any unselected lidar depth point ,satisfy: ； in Represents Euclidean distance. , , express for The complement; S23, for distance pixels Each lidar depth point at different distances The corresponding weights are calculated using the following formula. : ； ； in It is a constant; For all weight Normalization is performed to obtain each Final weight Used for k lidar depth points depth Perform a weighted summation to obtain a dense depth map. Each pixel diffusion depth : ； S24. Dense depth map Through camera intrinsic parameter matrix and camera external parameters Projecting back into the world coordinate system yields the depth point cloud corresponding to the current image. ; The calculation steps for the final loss in step S4 are as follows: S41. Transform the 3D Gaussian model in each training iteration using camera extrinsics. Projecting this onto the camera coordinate system yields the rendering depth for the current iteration. Where H represents the pixel height of the image and W represents the pixel width of the image; S42. Calculate the dense depth loss of the image using the following formula: ； in These are image pixel coordinates. Represents the L1 norm; S43. Record the dense depth loss for each image in the training dataset to obtain the dense depth loss vector. ,in This represents the number of images in the training set; for dense depth loss vectors The loss coefficient is obtained after normalization. : ； in and These are two control constants. This represents the Sigmoid normalization operation; S44. Calculate the image reconstruction loss for the i-th image in the training dataset. SSIM loss Then, with the dense depth loss in step S42 Together they constitute the overall loss, and the loss coefficient is also included. The i-th element Applying this to the overall loss, we obtain the final loss for the i-th image in the training dataset: ； in and These are adjustable weight hyperparameters.

2. The method for reconstructing a 3D Gaussian city scene based on point cloud depth diffusion according to claim 1, characterized in that, The training dataset in step S1 includes multiple temporally and spatially continuous image samples collected from the target scene, and each image sample contains a corresponding camera intrinsic parameter matrix. Camera external parameters SfM point cloud and lidar point cloud ,in It is a rotation matrix. It is a translation vector.

3. The method for reconstructing a 3D Gaussian city scene based on point cloud depth diffusion according to claim 1, characterized in that, In step S3, when uniformly sampling the depth point cloud using the voxel selection method, the three-dimensional space of the depth point cloud is divided into a voxel grid according to a preset voxel size. One depth point at the center of each voxel replaces all the depth point cloud inside the voxel, thereby obtaining a uniformly sampled point cloud.

4. The method for reconstructing a 3D Gaussian city scene based on point cloud depth diffusion according to claim 1, characterized in that, When uniformly sampling the depth point cloud using a voxel selection method, the depth point cloud is controlled by adjusting the voxel size. The density of midpoint cloud sampling.

5. The method for reconstructing a 3D Gaussian city scene based on point cloud depth diffusion according to claim 1, characterized in that, Weight values ​​in step S43 =2 and =100; Weight value in step S44 =0.2 and =0.

01.

6. The method for reconstructing a 3D Gaussian city scene based on point cloud depth diffusion according to claim 1, characterized in that, In step S5, rasterization technology is used to visualize the trained 3D Gaussian model.

7. A 3D Gaussian city scene reconstruction system based on point cloud depth diffusion, characterized in that, Includes the following modules: The data acquisition module is used to acquire a training dataset of the target scene for reconstructing the scene model. The training dataset contains a spatiotemporally continuous image sequence of the target scene, corresponding camera parameters, and LiDAR point cloud and SfM point cloud data. The depth point cloud generation module projects sparse LiDAR point clouds onto the image space of each image in the image sequence. It traverses the image space, treating each pixel as a target pixel, and selects the depths of multiple nearest neighbor point clouds for each target pixel. It calculates weights based on the distance between each point cloud and the target pixel, and then sums the depths of the selected point clouds according to the calculated weights to obtain the depth value of the target pixel. After traversing all pixels in the image space, it obtains the dense depth map corresponding to the image. The dense depth map is then projected onto the world coordinate system using camera pose parameters to obtain the depth point cloud. The initialization position generation module is used to uniformly sample the depth point cloud using a voxel selection method, and to combine the sampled point cloud with the SfM point cloud and lidar point cloud in the dataset to form a dense point cloud and add it to the training dataset as the initial position of the 3D Gaussian, so as to balance modeling speed and modeling quality. The model training module is used to train a 3D Gaussian model using the training dataset. During the training process, an additional dense depth loss function is added to the depth rendering result. The final loss is the weighted sum of image reconstruction loss, SSIM loss and dense depth loss multiplied by the loss coefficient. The dense depth map is input and the 3D Gaussian model is supervised and optimized based on the final loss to improve the modeling quality of the 3D Gaussian model for the scene. The visualization module is used to visualize the trained 3D Gaussian model and obtain explicit modeling results.

8. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the three-dimensional Gaussian city scene reconstruction method based on point cloud depth diffusion as described in any one of claims 1 to 6.

Images