Three-dimensional reconstruction method, device and equipment based on 3D Gaussian

Abstract

The invention discloses a three-dimensional reconstruction method, device and equipment based on 3D Gaussian. The method comprises the steps of performing data preprocessing based on a three-dimensional point cloud output by sparse reconstruction of a target scene, a camera internal reference and a camera pose; initializing and training a 3D Gaussian model based on a real image, the three-dimensional point cloud after candidate object mask screening, a camera pose, a depth information graph and a scale alignment parameter to obtain a coarse-grained 3D Gaussian model of the target scene; generating a dynamic target mask by taking a rendered image generated by rendering the target scene based on the coarse-grained 3D Gaussian model as a reference and combining the real image and the candidate object mask; and on the basis of the coarse-grained 3D Gaussian model, the dynamic target mask and all input data obtained by initializing and training the coarse-grained 3D Gaussian model, carrying out refined reconstruction and optimization on the coarse-grained 3D Gaussian model. According to the method, unification of high precision, high efficiency and high robustness is realized, and the increasing actual industrial application requirements are met.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D reconstruction technology, and in particular to a 3D reconstruction method, apparatus and device based on 3D Gaussian. Background Technology

[0002] For a long time, 3D reconstruction has been a key technology in computer vision, computer graphics, and virtual reality, with wide applications in digital twins, virtual reality, and autonomous driving. Traditional methods mainly rely on geometric modeling and multi-view stereo matching. However, these methods are generally highly dependent on scene texture, and their reconstruction accuracy is limited in areas with sparse or repetitive textures. At the same time, they have high computational complexity and poor adaptability to lighting changes, complex occlusion, and non-Lambertian surfaces, making it difficult to simultaneously meet the requirements of high precision and high efficiency for industrial applications.

[0003] In recent years, emerging technologies based on implicit neural representations, such as Neural Radiation Field (NeRF), have brought about groundbreaking changes to the field of 3D reconstruction. In particular, 3D Gaussian Splatting (3D-GS) has rapidly emerged as a revolutionary explicit scene representation method. 3D-GS utilizes a large number of learnable 3D Gaussian primitives to directly represent the scene, with each primitive possessing attributes such as position, covariance (controlling scale and rotation), opacity, and spherical harmonic function coefficients. Compared to the time-consuming network inference required by implicit methods like NeRF, 3D-GS achieves efficient differentiable rendering through a tile-based rasterizer, resulting in orders of magnitude improvements in training speed and real-time rendering frame rate, while maintaining excellent visual quality and geometric detail. It effectively reconciles the contradiction between reconstruction quality and computational efficiency, and has become a cutting-edge and highly promising technological direction.

[0004] Despite the significant advantages of 3D Gaussian spatter technology, its inherent technical framework still faces a series of inherent bottlenecks and challenges when dealing with complex and diverse real-world application scenarios, limiting its robustness, accuracy, and versatility. Existing technical solutions mainly suffer from the following shortcomings:

[0005] (1) The parameter initialization and optimization strategies have defects, which affect the reconstruction accuracy and robustness:

[0006] Coarse initialization: Existing methods typically rely on random or simple rules to initialize the position and size of Gaussian elements from sparse point clouds (such as those obtained through motion reconstruction techniques), lacking prior guidance on the global geometry of the scene. This coarse initialization can easily lead to redundant or unevenly distributed Gaussians in subsequent optimizations, affecting reconstruction efficiency and model compactness.

[0007] The optimization objective is singular: the optimization process heavily relies on pixel-level color differences between the rendered image and the input image (such as L1 or L2 loss). This single-level projection error constraint is highly susceptible to getting stuck in local optima in regions with sparse viewpoints, repetitive textures, or weak textures, leading to artifacts, geometric noise (such as "floating Gaussians"), or uneven surfaces. Furthermore, the loss function lacks multi-dimensional constraints on geometric consistency and distribution regularity, resulting in insufficient robustness of the model to the quality of the input data.

[0008] (2) There is a serious lack of ability to reconstruct dynamic scenes:

[0009] Existing 3D-GS frameworks and their mainstream improvements are primarily designed for static scenes. When moving objects exist in the scene, the model treats all observation data as the same static moment, leading to the incorrect fusion of different positional information along the motion trajectory into the same set of Gaussian parameters. This inevitably produces motion blur, ghosting, and geometric distortion in the reconstruction results, failing to achieve clear, temporally consistent 3D reconstruction of dynamic targets. Existing methods lack mechanisms for modeling and estimating the temporal dimension.

[0010] (3) Insufficient adaptability to large-scale scenarios and complex conditions:

[0011] Large-scale scene processing is inefficient: When 3D-GS is directly applied to urban or large-scale outdoor scenes, the number of Gaussian primitives required increases dramatically, leading to an explosive increase in memory usage and a sharp decline in training and rendering efficiency. Existing block or hierarchical strategies are often rigid, easily creating seams between blocks or resulting in insufficient utilization of global information, making it difficult to maintain consistent reconstruction accuracy over a wide area.

[0012] Weak or missing preprocessing steps: Many solutions neglect the crucial role of data preprocessing. For example, failure to accurately restore the true scale and coordinate system of the scene leads to distorted reconstructed model dimensions; in dynamic scenes, failure to effectively separate or identify dynamic targets from static backgrounds introduces insurmountable noise and ambiguity into subsequent reconstruction.

[0013] Unable to balance high precision, high efficiency, and high robustness: In summary, existing 3D-GS-based technologies have not yet systematically addressed multiple challenges, including sparse input viewpoints, drastic lighting changes, extreme scale variations (indoor / outdoor), and dynamic content. While pursuing real-time rendering speed, they often sacrifice the ability to finely reconstruct complex scenes; conversely, attempting to enhance detail may introduce excessive computational overhead and stability risks. Summary of the Invention

[0014] In order to fundamentally improve the initialization and optimization mechanism of Gaussian primitives, enhance their ability to represent dynamic scenes, and design efficient and robust strategies to cope with reconstruction tasks under large-scale and complex conditions, thereby truly achieving a unity of high precision, high efficiency, and high robustness to meet the growing needs of practical industrial applications, this invention is made.

[0015] As one aspect of the present invention, embodiments of the present invention provide a 3D reconstruction method based on 3D Gaussian, which may include:

[0016] Data preprocessing is performed on the 3D point cloud, camera intrinsics, and camera pose output from the sparse reconstruction of the target scene. The data preprocessing includes: distortion correction and downsampling of the visual data of the target scene based on the camera intrinsics to obtain a real image; target object segmentation and depth estimation processing are performed on the real image to generate candidate object masks and a depth information map of the target scene; scale restoration processing is performed based on the 3D point cloud, the camera pose, the candidate object masks, and the depth information map to determine scale alignment parameters.

[0017] Based on the real image, the 3D point cloud after candidate object masking, the camera pose, the depth information map, and the scale alignment parameters, a 3D Gaussian model is initialized and trained to obtain a coarse-grained 3D Gaussian model of the target scene.

[0018] Based on the rendered image generated by rendering the target scene using the coarse-grained 3D Gaussian model, a dynamic target mask is generated by combining the real image and the candidate object mask.

[0019] Based on the coarse-grained 3D Gaussian model, the dynamic target mask, and all input data initialized and trained by the coarse-grained 3D Gaussian model, the coarse-grained 3D Gaussian model is refined and optimized.

[0020] As another aspect of the present invention, embodiments of the present invention provide a 3D reconstruction device based on 3D Gaussian, which may include:

[0021] A data preprocessing module is used to perform data preprocessing based on the 3D point cloud, camera intrinsic parameters, and camera pose output from the sparse reconstruction of the target scene. The data preprocessing includes: performing distortion correction and downsampling on the visual data of the target scene based on the camera intrinsic parameters to obtain a real image; performing target object segmentation and depth estimation on the real image to generate candidate object masks and a depth information map of the target scene; and performing scale restoration processing based on the 3D point cloud, the camera pose, the candidate object masks, and the depth information map to determine scale alignment parameters.

[0022] The coarse model reconstruction module is used to initialize and train a 3D Gaussian model based on the real image, the 3D point cloud after the candidate object mask is filtered, the camera pose, the depth information map, and the scale alignment parameters, so as to obtain a coarse-grained 3D Gaussian model of the target scene.

[0023] The dynamic mask generation module is used to generate a dynamic target mask based on the rendered image generated by rendering the target scene using the coarse-grained 3D Gaussian model, combined with the real image and the candidate object mask.

[0024] The fine-grained reconstruction module is used to perform fine-grained reconstruction and optimization of the coarse-grained 3D Gaussian model based on the coarse-grained 3D Gaussian model, the dynamic target mask, and all input data initialized and trained by the coarse-grained 3D Gaussian model.

[0025] As another aspect of the present invention, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described 3D Gaussian-based three-dimensional reconstruction method.

[0026] As another aspect of the present invention, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described 3D Gaussian-based three-dimensional reconstruction method.

[0027] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0028] This invention provides a 3D reconstruction method, apparatus, and device based on 3D Gaussian. This method systematically solves three core problems existing in current 3D Gaussian splicing (3D-GS) based 3D reconstruction methods through a co-designed process of "data preprocessing - coarse-grained modeling - dynamic mask generation - fine-grained reconstruction": First, by integrating innovative preprocessing steps such as distortion correction and object-aware scale recovery, accurate and geometrically aware multi-source supervision signals are provided for the initialization of the Gaussian model, improving the accuracy of parameter optimization from the source and effectively suppressing artifacts and edge blurring; Second, the method creatively utilizes the trained coarse-grained 3D Gaussian model as a static... The geometric benchmark of the dynamic scene is used to adaptively generate dynamic target masks by comparing its rendered image with the real input image and combining it with candidate object masks. This isolates dynamic interference in the fine reconstruction stage and fundamentally solves the problems of ghosting and structural disorder in dynamic scene reconstruction. Finally, the scheme combines robust global coarse reconstruction with optional local fine optimization, so that the reconstruction process can flexibly adapt to different needs from small indoor scenes to large outdoor scenes. While ensuring high-precision modeling, it significantly improves the overall robustness and practicality of the algorithm for complex, dynamic and large-scale scenes, and achieves an effective unity of high precision, high efficiency and high robustness in 3D reconstruction tasks.

[0029] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0030] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0031] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0032] Figure 1 This is a flowchart of the 3D reconstruction method based on 3D Gaussian provided in Embodiment 1 of the present invention;

[0033] Figure 2 This is a flowchart illustrating the determination of scale alignment parameters provided in Embodiment 1 of the present invention.

[0034] Figure 3 This is a flowchart illustrating the coarse model reconstruction process in step S12 of Embodiment 1 of the present invention.

[0035] Figure 4 This is a flowchart illustrating the specific process of generating the dynamic target mask in step S13 of Embodiment 1 of the present invention.

[0036] Figure 5 This is a schematic diagram of the structure of the 3D reconstruction device based on 3D Gaussian provided in Embodiment 1 of the present invention;

[0037] Figure 6 The flowchart below shows the detailed three-dimensional reconstruction method based on 3D Gaussian provided in Embodiment 2 of the present invention.

[0038] Figure 7 This is a flowchart illustrating the process of determining the associated image for each sub-block, as provided in Embodiment 2 of the present invention.

[0039] Figure 8 This is a schematic diagram of the structure of the 3D reconstruction device based on 3D Gaussian provided in Embodiment 2 of the present invention. Detailed Implementation

[0040] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0041] Example 1

[0042] Embodiment 1 of this invention provides a 3D reconstruction method based on 3D Gaussian, referring to... Figure 1 As shown, the method may include the following steps:

[0043] Step S11: Perform data preprocessing based on the 3D point cloud, camera intrinsics, and camera pose output from the sparse reconstruction of the target scene. Data preprocessing may include: performing distortion correction and downsampling on the visual data of the target scene based on the camera intrinsics to obtain a real image; performing target object segmentation and depth estimation on the real image to generate candidate object masks and a depth information map of the target scene; and performing scale restoration processing based on the 3D point cloud, camera pose, candidate object masks, and depth information map to determine the scale alignment parameters.

[0044] Step S11 plays a crucial role in deeply processing the results of sparse reconstruction, aiming to generate the foundational data required for 3D-GS reconstruction. Extensive practical experience and professional verification have shown that accurate initialization plays a decisive role in the quality of 3D-GS reconstruction. Addressing the significant deficiencies or insufficient refinement in the initial data preprocessing stage of existing technologies, the inventors, through in-depth analysis and summarization of these lessons learned, and fully considering the application requirements and characteristics of this embodiment, have meticulously designed a targeted preprocessing workflow.

[0045] This step, through a series of rigorous and targeted preprocessing steps, can maximize the extraction and optimization of effective information in the sparse reconstruction results, providing more accurate and reliable basic data for subsequent 3D-GS reconstruction, thereby significantly improving the final reconstruction quality and effect, and ensuring that the reconstruction results are more in line with the needs of actual application scenarios.

[0046] In this embodiment, the scale alignment parameters are the scale factor and offset factor used to align the relative scale in the depth information map to the real physical scale.

[0047] Step S12: Based on the real image, the 3D point cloud after candidate object masking, camera pose, depth information map, and scale alignment parameters, initialize and train a 3D Gaussian model to obtain a coarse-grained 3D Gaussian model of the target scene.

[0048] In this embodiment, the "3D Gaussian model" is a general 3D Gaussian model, referring to a coarse-grained or fine-grained static 3D Gaussian model constructed through initialization and training in 3D Gaussian-based 3D reconstruction methods. This model represents the static structure of the target scene (i.e., the non-dynamic, fixed background portion). Subsequent steps eliminate interference from dynamic targets (such as pedestrians and vehicles), focusing on reconstructing the static geometric structure and appearance information of the scene, providing a stable spatial benchmark and geometric reference for subsequent dynamic target detection, fine-grained reconstruction, and scene alignment.

[0049] Step S13: Based on the rendered image generated by rendering the target scene using a coarse-grained 3D Gaussian model, a dynamic target mask is generated by combining the real image and the candidate object mask.

[0050] Step S14: Based on all the input data of the coarse-grained 3D Gaussian model, dynamic target mask, and initialization and training of the coarse-grained 3D Gaussian model, refine and optimize the coarse-grained 3D Gaussian model.

[0051] The 3D Gaussian-based 3D reconstruction method provided in this embodiment of the invention systematically solves three core problems existing in current 3D Gaussian splicing (3D-GS)-based 3D reconstruction methods through a co-designed "data preprocessing - coarse-grained modeling - dynamic mask generation - fine-grained reconstruction" process: First, by integrating innovative preprocessing steps such as distortion correction and object-aware scale recovery, accurate and geometrically aware multi-source supervision signals are provided for the initialization of the Gaussian model, improving the accuracy of parameter optimization from the source and effectively suppressing artifacts and edge blurring; Second, the trained coarse-grained 3D Gaussian model is creatively used as a static scene... The geometric benchmark, by comparing its rendered image with the real input image and combining it with candidate object masks, can adaptively generate dynamic target masks, thereby isolating dynamic interference in the fine reconstruction stage and fundamentally solving the problems of ghosting and structural disorder in dynamic scene reconstruction. Finally, the scheme combines robust global coarse reconstruction with optional local fine optimization, enabling the reconstruction process to flexibly adapt to different needs from small indoor scenes to large outdoor scenes. While ensuring high-precision modeling, it significantly improves the overall robustness and practicality of the algorithm for complex, dynamic, and large-scale scenes, achieving an effective unity of high precision, high efficiency, and high robustness in 3D reconstruction tasks.

[0052] In one embodiment, the data preprocessing in step S11 above includes operations such as distortion correction, downsampling, target object segmentation, depth estimation, and scale restoration, which are specifically implemented as follows:

[0053] (1) Distortion correction and downsampling

[0054] The visual data of the target scene is distorted and downsampled based on camera intrinsic parameters to obtain a realistic image. It should be noted that the distortion correction of the original visual data (image or video) based on camera intrinsic parameters (optical properties, focal length, principal point, etc.) in this embodiment is a well-known technique. This embodiment addresses the differences in performance between different devices and the varying requirements for reconstruction results. The inventors dynamically set the downsampling scale according to the actual situation (modeling accuracy, device performance, project requirements, etc.). This flexible approach finds the optimal balance between accuracy and efficiency, ensuring a significant improvement in data processing efficiency without sacrificing excessive reconstruction accuracy, thereby achieving the desired reconstruction effect. For example, for high-performance devices, the downsampling scale can be appropriately reduced to retain more detail information; while for relatively weaker devices, the downsampling scale can be increased to speed up processing.

[0055] (2) Target object segmentation

[0056] The real image is segmented to generate a candidate object mask. This step is to perform target object segmentation on the real image obtained by distortion correction and downsampling in step (1) to generate a candidate object mask.

[0057] Based on extensive theoretical research and practical experience, the inventors discovered that moving targets in a scene can severely impact accurate scene reconstruction, often causing artifacts, floating Gaussians, and a series of other problems. To effectively address these issues, this invention fully considers the characteristics of its application scenario and pre-defines several common object categories that may generate moving targets, including pedestrians, bicycles, cars, vans, trucks, tricycles, buses, and motorcycles. To accurately identify all possible moving targets in the acquired images, this embodiment adopts a global recognition strategy, striving to comprehensively detect every category, regardless of whether the target is stationary or moving, avoiding the omission of any potential moving targets. To this end, this embodiment carefully designs a two-step method to generate accurate candidate object masks.

[0058] The first step involves employing an object detection model. This embodiment of the invention utilizes an advanced YOLOv11 pre-trained detection model, and has undergone in-depth model optimization for specific application scenarios. This optimized model accurately obtains precise bounding boxes for each object category, providing crucial localization information for subsequent, more accurate segmentation operations.

[0059] The second step involves using SegmentAnything (a deep learning-based image segmentation model) for target segmentation. The detection bounding boxes obtained in the first step are used as prompts input into the segmentation model to obtain more accurate segmentation results. Furthermore, considering that subtle errors may exist during detection or segmentation, such as blurred target boundaries or missing features, which could prevent the target region from being completely contained within the mask, this embodiment expands the segmentation mask (i.e., the candidate object mask) by a certain proportion to ensure that the mask completely covers the target region, minimizing potential boundary omissions.

[0060] (3) Depth estimation

[0061] This embodiment performs depth estimation processing on the real image to generate a depth information map of the target scene; this step is to perform depth estimation on the real image obtained by distortion correction and downsampling processing in step (1) to generate a depth information map of the target scene.

[0062] Through extensive research and experimental verification, the inventors discovered that effective depth supervision plays a crucial role in 3D scene reconstruction, especially for 3D-GS-based reconstruction methods. Accurate restoration of scene structure is a key challenge in 3D Gaussian reconstruction, particularly when the scene contains regions with indistinct features, repetitive textures, or occlusion. A lack of accurate depth information can lead to serious deviations in the reconstruction results, such as artifacts, inaccurate Gaussian distribution, and scene scale distortion. Introducing depth supervision significantly improves these issues. It provides crucial reference for the distribution, fusion, and optimization of 3D Gaussian, enabling Gaussian to more accurately simulate the real geometric structure of the scene, thereby enhancing the realism and accuracy of the reconstructed scene.

[0063] Based on this important understanding, this embodiment of the invention performs depth estimation on the image after distortion correction and downsampling during the data preprocessing stage. To ensure the accuracy and efficiency of depth estimation, this embodiment of the invention selects the advanced DepthAnything model. This model, with its powerful feature extraction and depth prediction capabilities, can accurately infer the depth information of a scene from an image, performing excellently even in complex environments. After obtaining the depth estimation result, considering the consistency of subsequent processing and data processing efficiency, this embodiment performs downsampling processing on it with the same size factor as the previous downsampling operation.

[0064] (4) Scale restoration

[0065] Depth estimation only presents a relative scale and lacks absolute scale information. However, scale recovery is essential for effective supervision during scene reconstruction. Both sparse reconstruction and depth estimation can introduce errors, which, if not effectively eliminated, will severely interfere with the accurate calculation of scale and offset factors, thus negatively impacting subsequent reconstruction work. Therefore, this invention incorporates a targeted error elimination strategy based on sparse reconstruction results to perform scale recovery, aiming to accurately calculate scale and offset factors. This invention performs scale recovery processing based on 3D point clouds, camera pose, candidate object masks, and depth information maps to determine scale alignment parameters.

[0066] Reference Figure 2 As shown, determining the scale alignment parameters in this embodiment may specifically include the following steps:

[0067] Step S23: Based on the correspondence between the candidate object mask and the 3D point cloud and image features, select the valid 3D points that are not covered by the candidate object mask.

[0068] This step filters 3D points based on candidate object masks, by obtaining each 3D point. With each image Matching relationships between points on a 3D point In the image The corresponding two-dimensional point on is The coordinates here clearly define the original detection location of the 3D point in the image. In the previous object segmentation step, this implementation has already obtained the coordinates for each image. Candidate object mask Object mask It is an image A matrix of the same size, where the element values ​​in each object region are labeled as 1~m, and the element values ​​in other regions are 0. This represents the value of the element in the u-th row and v-th column of the mask matrix. It is used to determine the value of two-dimensional points. In candidate object mask The value at the corresponding position (a value greater than 0 indicates the presence of a target object) is used to determine whether to exclude the 3D point.

[0069] Step S24: Based on the camera pose, transform the effective 3D points to the camera coordinate system and calculate their inverse depth.

[0070] This step involves transforming the selected valid 3D points to the camera coordinate system. That is: ;

[0071] Then calculate the inverse depth of the 3D points after transformation to the camera coordinate system, i.e.: .

[0072] The reason for choosing to extract inverse depth in this embodiment is that it better addresses the distribution characteristics of depth values ​​when processing depth information, especially in scenes with large depth variations. Inverse depth can make the data distribution more uniform, thereby improving the stability and accuracy of the calculation of scale and offset factors. Mathematically, inverse depth can transform nonlinear changes in depth values ​​into relatively linear changes, facilitating subsequent numerical calculations and optimization.

[0073] Step S25: Based on the coordinates of the two-dimensional feature points in the image corresponding to the effective three-dimensional points, extract the corresponding inverse depth values ​​from the depth information map.

[0074] This step extracts the corresponding inverse depth value (invmonodepth) from the depth estimation results based on the two-dimensional point information corresponding to the valid three-dimensional points. During the extraction process, bilinear interpolation is used based on specified position coordinates to accurately obtain the inverse depth value.

[0075] Step S26: Based on the inverse depth set of effective 3D points and the inverse depth value set extracted from the depth information map, calculate the scale factor and offset factor, and use the scale factor and offset factor as scale alignment parameters.

[0076] In this step, the inverse depth median of the sparse reconstruction valid points in the frame is calculated respectively. and mean absolute deviation It also calculates the median of the inverse depth of the monocular depth estimation. and mean absolute deviation Finally, the scaling factor is calculated. and offset factor .

[0077] This embodiment effectively addresses the core issues of geometric structure deviation and insufficient reconstruction accuracy caused by inconsistent depth estimation scales in existing 3D Gaussian reconstruction techniques through multi-step collaborative processing. Specifically, the method first utilizes candidate object masks and the correspondence between 3D point clouds and image features to accurately filter out effective 3D points that only reflect static background structures, actively eliminating noise interference from dynamic targets. Subsequently, by transforming the effective 3D points to the camera coordinate system and calculating their inverse depth, and extracting the inverse depth values ​​of the corresponding coordinates from the depth information map, two sets of depth values ​​with comparable distributional statistical features are constructed. Finally, the scale factor and offset factor calculated based on these two sets serve as precise scale alignment parameters, achieving robust alignment from the relative scale of monocular depth estimation to the absolute scale of sparse reconstruction. This technical effect not only ensures the accuracy of geometric constraints during subsequent 3D Gaussian model initialization and training, improving the global structural consistency of the reconstruction model, but also enhances the adaptability to real scenes containing dynamic targets and complex textures through refined data processing, laying a crucial scale benchmark for high-precision 3D reconstruction.

[0078] In one embodiment, reference is also made to Figure 2 As shown, before step S23, which filters valid 3D points, the following steps may also be included:

[0079] Step S21: Calculate the average reprojection error of each 3D point on its matched associated image. The reprojection error reflects the degree of deviation between the projected position of the 3D point on the image and its actual observed position. For each 3D point... Its average reprojection error The calculation formula is: .in, Representing a three-dimensional point The reprojection error on the j-th related image reflects the difference between the theoretical and actual observed positions of the 3D point projected onto the j-th image; n is the reprojection error relative to the 3D point. The number of images that have a matching relationship.

[0080] Step S22: Exclude 3D points whose average reprojection error is greater than a preset threshold.

[0081] In this embodiment, after calculating the average reprojection error, an error threshold T is set. When a certain 3D point... Average reprojection error If the error is large, it indicates that the 3D point may be due to incorrect matching or other interference factors, and its reliability is low. This implementation will exclude such 3D points with large errors and prevent them from participating in subsequent scale recovery and other operations.

[0082] This embodiment uses a method based on the average reprojection error of 3D points for screening. By calculating and excluding 3D points with reprojection errors greater than a preset threshold, it can effectively identify and eliminate abnormal 3D points caused by feature mismatch, image noise, or inaccurate camera parameter estimation. This significantly improves the accuracy and geometric consistency of the 3D point set used to calculate the scale factor in the subsequent scale restoration step, ensuring the reliability of the conversion benchmark from monocular depth estimation to real-world scale, and thus improving the geometric accuracy and global consistency of the entire 3D reconstruction model from the source.

[0083] In one embodiment, in the process of large-scale scene reconstruction, the reconstruction and training of the coarse model is a crucial link between data preprocessing and subsequent fine modeling. Its core value lies in rapidly constructing a globally consistent scene framework at a controllable computational cost, providing a stable data benchmark and logical support for subsequent fine modeling. Specifically, on the one hand, the coarse model completes Gaussian initialization and rapid optimization based on preprocessed multi-source data, accurately outlining the core geometric structure of the scene (such as building outlines, road directions, terrain undulations, etc.) from a global perspective, establishing a global spatial coordinate benchmark. On the other hand, the global structure of the coarse model provides crucial data support for the identification of dynamic objects. Through a carefully designed dynamic target identification program, the dynamic and static attributes of target objects can be effectively distinguished. As a key data input in the fine modeling stage, it provides the necessary guarantee for eliminating dynamic target interference and obtaining globally accurate geometric structure and scene realism in the fine modeling stage.

[0084] Reference Figure 3 As shown, the coarse model reconstruction in step S12 above may specifically include the following steps:

[0085] Step S31: Initialize the 3D Gaussian model based on the filtered 3D point cloud and camera pose, and constrain the scale parameter of each Gaussian element perpendicular to the normal direction of the scene surface to zero to form a planar Gaussian representation; the parameters of each Gaussian element include: three-dimensional spatial coordinates (xyz, corresponding to the spatial position of the Gaussian center), scale parameter (scales, describing the extension range of the Gaussian in three dimensions), rotation parameter (rotations, defining the spatial orientation of the Gaussian ellipsoid), spherical harmonic function coefficients (DC component shs_dc and higher-order component shs_rest, jointly encoding the surface color and lighting response characteristics), and opacity parameter (opacities, controlling the visibility of the Gaussian during rendering).

[0086] It should be noted that the embodiments of this invention have made targeted designs for the 3D Gaussian model during the initialization phase. The dimension of the scale parameter perpendicular to the scene surface normal vector is fixed to 0, and only the two scale dimensions along the surface tangent direction are retained, thus forming the characteristics of "2D surface Gaussian / planar Gaussian". The advantages of this design are reflected in two aspects. First, it is more in line with the geometric essence of the scene. For example, the core structure of large-scale urban scenes is mostly two-dimensional surface (such as building facades, road planes, roofs, etc.). By constraining one dimension of the scale to 0, the Gaussian model can naturally fit the surface geometric characteristics, avoiding the "volume expansion" problem caused by redundant dimensions in traditional 3D volumetric Gaussian models (such as over-covering the space inside and outside the surface), thereby capturing the geometric structure and boundary features of the scene more accurately. Second, by reducing the parameter degree of freedom of one scale dimension, the parameter update overhead can be reduced in large-scale scenes, while reducing the memory usage and improving efficiency.

[0087] Step S32: During the training process, the current 3D Gaussian model is rendered based on the real image, depth information map, and scale alignment parameters to obtain the rendered image and the rendered depth map.

[0088] Step S33: Calculate the composite loss function; wherein, the composite loss function includes a color loss term, a depth regularization loss term, and a normal consistency loss term; the color loss term is the weighted loss between the rendered image and the corresponding real image, including the L1 loss component and the structural similarity loss component; the depth regularization loss term is the weighted loss between the rendered depth map and the scaled depth map obtained based on the depth information map and the scale alignment parameter transformation, including the L1 loss component and the structural similarity loss component; the normal consistency loss term is used to constrain the consistency between the rendered surface normal and the geometric normal represented by the coarse-grained 3D Gaussian model.

[0089] During scene training, to simultaneously ensure the model's visual realism and geometric accuracy, an optimization method combining multiple loss functions is employed, as detailed below:

[0090]

[0091]

[0092]

[0093]

[0094] The color loss (RGB loss) uses a combination of L1 loss and SSIM loss to constrain the visual consistency between the Gaussian model-rendered image and the real image. L1 loss constrains the global accuracy of color, effectively avoiding overall tone shift, while SSIM integrates structural similarity indicators such as brightness and contrast to constrain the local structure and texture details of the image, compensating for the deficiency of L1 loss, which only focuses on pixel value differences and ignores structural relationships. The depth regularization loss uses a combination of L1 loss and SSIM loss to constrain the visual consistency between the depth rendered by the Gaussian model and the preprocessed estimated depth. It can ensure the accuracy of the global geometric scale through L1 loss and the continuity of the local depth structure through SSIM, so that the 3D position of the Gaussian model fits the real scale of the scene and restores the geometric topology of the scene.

[0095] Step S34: Update the parameters of the 3D Gaussian model using a dynamic gradient optimization strategy; wherein the gradient weight of each Gaussian element is determined based on its projected area, and the gradient scaling factor is determined based on the ratio of its depth to the average depth of the scene.

[0096] Due to the large depth span and significant scale differences in large-scale scenes, especially urban scenes, and issues such as insufficient input data viewpoints or repetitive textures, Gaussian artifacts and floating artifacts frequently occur. To further improve training stability and parameter optimization accuracy, and effectively suppress floating Gaussian artifacts, targeted optimizations were made to gradient calculation and propagation during the optimization process. Through strategies such as dynamic weighted gradients and scaled gradients, Gaussian artifacts are preferentially generated in texture-rich or edge regions. Furthermore, the original gradients of near-field Gaussian artifacts are prioritized to ensure detail optimization, while far-field Gaussian gradients are suppressed to avoid interfering with fine near-field structures. The specific calculation formula is as follows:

[0097]

[0098]

[0099] In the scaling factor formula above, the current Gaussian depth may be very small (even close to 0). If the denominator is zero or extremely small, the scaling factor will approach infinity, leading to unstable numerical calculations. Therefore, It is a very small positive number (e.g., 10). -8 Or 10 -6This technique is used to ensure that the denominator is not zero without affecting the calculation results of a Gaussian with a large depth. It is a common technique in machine learning and numerical optimization, especially widely used when dealing with inverse depth or scale calculations. Its functions are: firstly, to prevent division by zero errors; secondly, to smooth gradient calculations and avoid gradient explosion caused by excessively small depths; and thirdly, to maintain numerical stability during optimization, making training more stable.

[0100] Step S35: Repeat the above training steps, composite loss function calculation steps, and update steps until the preset training stopping condition is met to obtain a coarse-grained 3D Gaussian model.

[0101] The embodiments of the present invention systematically solve the inherent defects of existing 3D Gaussian methods in parameter initialization, single optimization objective and training stability in the coarse-grained model reconstruction stage by introducing planar Gaussian representation and multi-level composite loss constraint mechanism, and with the help of dynamic gradient optimization strategy. Specifically, this embodiment constrains the Gaussian scale parameter during initialization, forcing Gaussian primitives to conform to the scene surface, effectively suppressing the "volume expansion" problem. This provides a more compact and realistic explicit representation for large-scale scenes, significantly improving the geometric accuracy and model efficiency of the reconstruction. During the training and optimization phase, color loss, depth regularization loss, and normal consistency loss are fused to construct a comprehensive constraint on visual appearance, geometric scale, and surface continuity. This overcomes the limitations of traditional methods that rely solely on single-level projection errors, greatly enhancing the model's robustness in sparse or repetitive texture regions and effectively reducing artifacts and "floating Gaussians." Furthermore, the dynamic gradient optimization strategy adaptively adjusts the optimization weights of Gaussian primitives based on their projection contribution and depth information, prioritizing the detailed depiction of near-field details while suppressing far-field noise interference, thus achieving rapid convergence and stable optimization during training. Overall, this embodiment lays a key technical foundation for obtaining high-quality, highly consistent coarse-grained scene models and provides a reliable global geometric benchmark for subsequent refined reconstruction and dynamic target processing.

[0102] In one embodiment, step S13 is the process of generating a dynamic target mask. Dynamic targets such as vehicles and pedestrians in the target scene can cause geometric deviations, edge artifacts, and global consistency violations in static scene reconstruction, severely affecting reconstruction accuracy. Therefore, a dynamic target mask generation method based on coarse model rendering comparison is designed, referring to... Figure 4 As shown, the following steps may be included:

[0103] Step S41: Calculate the RGB difference matrix and structural similarity matrix between the rendered image and the real image.

[0104] Step S42: Normalize and weight the RGB difference matrix and the structural similarity matrix to obtain a pixel-level combined loss heatmap.

[0105] The formula for the pixel-level combination loss heatmap in this embodiment is as follows:

[0106]

[0107] Step S43: Identify each target instance region based on the candidate object mask, and calculate the average loss value of the combined loss heatmap within each instance region.

[0108] Instance-level loss statistics and adaptive threshold determination: Based on the label_map, all instances are traversed, the average combined loss within each instance region is calculated, and the dynamic feature quantization value of each instance is obtained (the higher the loss, the greater the dynamic probability).

[0109] Step S44: Based on the distribution of average loss values ​​across all instance regions, an adaptive threshold is set. Instance regions with average loss values ​​higher than the adaptive threshold are identified as dynamic targets, and a dynamic target mask is generated. This step uses the heatmap calculated above, with the mean minus the standard deviation of the effective region as the threshold, to identify instance regions with combined losses higher than the threshold as dynamic targets, and generates a dynamic target mask.

[0110] This embodiment performs dual analysis of RGB differences and structural similarity between the coarse-grained 3D Gaussian model rendered image and the real input image. By utilizing candidate object masks to achieve instance-level difference statistics and adaptive threshold determination, it achieves accurate identification and mask generation of dynamic targets. This technical solution effectively overcomes the technical shortcomings of existing 3D Gaussian reconstruction methods, such as motion blur, ghosting, and structural distortion in dynamic scenes. It can accurately distinguish between static structures and moving objects in complex scenes, providing a reliable data foundation for subsequent refined reconstruction and significantly improving the geometric consistency and visual realism of 3D reconstruction results in dynamic environments.

[0111] Based on the same inventive concept, Embodiment 1 of the present invention also provides a 3D reconstruction device based on 3D Gaussian, referring to... Figure 5 As shown, the device may include: a data preprocessing module 51, a coarse model reconstruction module 52, a dynamic mask generation module 53, and a fine reconstruction module 54, and its working principle is as follows:

[0112] The data preprocessing module 51 is used to perform data preprocessing based on the 3D point cloud, camera intrinsic parameters, and camera pose output from the sparse reconstruction of the target scene. The data preprocessing includes: performing distortion correction and downsampling processing on the visual data of the target scene based on the camera intrinsic parameters to obtain a real image; performing target object segmentation and depth estimation processing on the real image to generate candidate object masks and depth information maps of the target scene; and performing scale restoration processing based on the 3D point cloud, camera pose, candidate object masks, and depth information maps to determine scale alignment parameters.

[0113] The coarse model reconstruction module 52 is used to initialize and train a 3D Gaussian model based on real images, 3D point clouds after candidate object mask screening, camera pose, depth information map, and scale alignment parameters to obtain a coarse-grained 3D Gaussian model of the target scene.

[0114] The dynamic mask generation module 53 is used to generate a dynamic target mask based on the rendered image generated by rendering the target scene using a coarse-grained 3D Gaussian model, combined with the real image and the candidate object mask.

[0115] The fine-grained reconstruction module 54 is used to perform fine-grained reconstruction and optimization of the coarse-grained 3D Gaussian model based on all input data that the coarse-grained 3D Gaussian model has been initialized and trained, including the coarse-grained 3D Gaussian model, dynamic target mask, and coarse-grained 3D Gaussian model.

[0116] This embodiment achieves high-precision and robust 3D reconstruction of large-scale scenes with a mix of static and dynamic elements through a modular hardware design. Specifically, the device systematically optimizes input data through a data preprocessing module, significantly improving the accuracy of initialization; it constructs a global geometric benchmark through a coarse model reconstruction module, providing a stable spatial reference for subsequent processing; it accurately identifies and isolates moving targets through a dynamic mask generation module, effectively eliminating artifacts and geometric distortions caused by dynamic interference; and finally, it optimizes this reliable foundation through a fine reconstruction module, ensuring that the final reconstructed model possesses both excellent detail reproduction capabilities and global consistency in complex real-world scenes (especially outdoor urban scenes). Thus, at the hardware level, the advantages of the method of this invention are solidified into a stable and efficient dedicated solution.

[0117] In one embodiment, the data preprocessing module 51 is specifically used for:

[0118] Based on the correspondence between the candidate object mask and the 3D point cloud and image features, valid 3D points not covered by the candidate object mask are selected.

[0119] Based on the camera pose, the effective 3D points are transformed to the camera coordinate system and their inverse depth is calculated;

[0120] Based on the coordinates of the two-dimensional feature points in the image corresponding to the effective three-dimensional points, the corresponding inverse depth values ​​are extracted from the depth information map.

[0121] Based on the inverse depth set of the effective 3D points and the inverse depth value set extracted from the depth information map, a scale factor and an offset factor are calculated, and the scale factor and the offset factor are used as the scale alignment parameters.

[0122] Specifically, the data preprocessing module 51 is also used for:

[0123] Calculate the average reprojection error of each 3D point on its matched associated image;

[0124] Three-dimensional points with an average reprojection error greater than a preset threshold are excluded.

[0125] In one embodiment, the coarse model reconstruction module 52 is specifically used for:

[0126] The 3D Gaussian model is initialized based on the selected 3D point cloud and camera pose, and the scale parameter of each Gaussian element perpendicular to the normal direction of the scene surface is constrained to zero to form a planar Gaussian representation. The parameters of each Gaussian element include: 3D spatial coordinates, scale parameter, rotation parameter, spherical harmonic function coefficient, and transparency parameter.

[0127] During training, the current 3D Gaussian model is rendered based on the real image, the depth information map, and the scale alignment parameters to obtain the rendered image and the rendered depth map.

[0128] Calculate a composite loss function; wherein the composite loss function includes a color loss term, a depth regularization loss term, and a normal consistency loss term; the color loss term is a weighted loss between the rendered image and the corresponding real image, including an L1 loss component and a structural similarity loss component; the depth regularization loss term is a weighted loss between the rendered depth map and the scaled depth map obtained based on the depth information map and the scale alignment parameter, including an L1 loss component and a structural similarity loss component; the normal consistency loss term is used to constrain the consistency between the rendered surface normal and the geometric normal represented by the coarse-grained 3D Gaussian model;

[0129] The parameters of the 3D Gaussian model are updated using a dynamic gradient optimization strategy; wherein the gradient weight of each Gaussian element is determined based on its projected area, and the gradient scaling factor is determined based on the ratio of its depth to the average depth of the scene.

[0130] Repeat the above training steps, composite loss function calculation steps, and update steps until the preset training stopping condition is met to obtain the coarse-grained 3D Gaussian model.

[0131] In one embodiment, the dynamic mask generation module 53 is specifically used for:

[0132] Calculate the RGB difference matrix and structural similarity matrix between the rendered image and the real image;

[0133] The RGB difference matrix and structural similarity matrix are normalized and weighted and fused to obtain a pixel-level combination loss heatmap;

[0134] Based on the candidate object mask, each target instance region is identified, and the average loss value of the combined loss heatmap within each instance region is calculated.

[0135] An adaptive threshold is set based on the distribution of the average loss value of all instance regions. Instance regions with an average loss value higher than the adaptive threshold are identified as dynamic targets, and the dynamic target mask is generated.

[0136] Based on the same inventive concept, this embodiment of the invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described 3D Gaussian-based three-dimensional reconstruction method.

[0137] This embodiment stores a computer program to implement the aforementioned 3D reconstruction method during execution, thus solidifying and materializing the innovative technical effects of this method into a standardized, reusable tangible product. This medium allows the high-precision, high-efficiency, and highly robust 3D reconstruction capabilities achieved by the method to be independent of specific computing devices and execution environments, enabling widespread deployment and convenient distribution of the technical solution. Users can reliably reproduce the significant technical effects of the method by loading and running the program in this medium, including: obtaining accurate geometric benchmarks through data preprocessing and scale recovery; improving reconstruction accuracy and training efficiency through planar Gaussian initialization and multi-constraint optimization; effectively eliminating dynamic interference through dynamic target masks based on coarse models; and efficiently processing large-scale scenes through optional block parallelism and dynamic pruning strategies. Therefore, this claim effectively expands the patent protection dimensions for the invention, transforming the core advantages of the method innovation into a software product that can be mass-produced and commercialized.

[0138] Based on the same inventive concept, this embodiment of the invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described 3D Gaussian-based three-dimensional reconstruction method.

[0139] This computer device executes a stored computer program to implement the aforementioned 3D Gaussian-based 3D reconstruction method, thereby achieving the following comprehensive technical effects: effectively improving the accuracy, efficiency, and robustness of 3D reconstruction of large-scale static scenes. Specifically, through innovative data preprocessing (scale recovery, dynamic candidate selection) and planar Gaussian initialization, the geometric accuracy and training stability of the 3D Gaussian model are significantly improved; through a two-stage reconstruction framework from coarse to fine, and by introducing dynamic target culling and block-based parallel optimization based on the coarse model, robust handling of dynamic disturbances and efficient, high-precision modeling of massive amounts of data are achieved, ultimately generating 3D reconstruction results with realistic scale, geographic alignment, and rich detail.

[0140] The principles by which the above-mentioned devices, media, and related equipment in the embodiments of the present invention solve the problem are similar to those of the aforementioned methods. Therefore, their implementation can refer to the implementation of the aforementioned methods, and repeated details will not be repeated.

[0141] Example 2

[0142] Embodiment 2 of this invention provides a detailed 3D reconstruction method based on 3D Gaussian, referring to... Figure 6 As shown, the method may include the following steps:

[0143] Step S61: Obtain scene data of the target scene. The scene data includes visual data and optionally GPS data. The visual data includes image data and / or video data.

[0144] In this embodiment, scene data for the target scene can be acquired using diverse devices, such as drones and mobile phones, to collect data on the predetermined scene. Data formats include, but are not limited to, images and videos. The predetermined scene can be large-scale, such as city panoramas or large industrial parks, or relatively small-scale, such as indoor living spaces or small sculptures; the scene's scope is unrestricted, and data can be collected. The collection of data is flexible in terms of whether or not it includes GPS data. It should be noted that GPS data is crucial in the entire 3D reconstruction process. It is the key foundation for restoring the scene's true physical scale, allowing for the accurate restoration of the size of each element in the reconstructed scene to a proportion consistent with reality. Simultaneously, GPS data is also the cornerstone for aligning the reconstructed scene to real-world geographic coordinates, enabling the reconstructed scene to find its precise corresponding location in real-world geographic space. This greatly improves the practicality and accuracy of the reconstructed scene, playing a vital role in applications such as geographic information systems and urban planning simulations.

[0145] Step S62: Perform sparse reconstruction based on scene data to generate camera intrinsic parameters, camera pose, 3D point cloud, and the correspondence between image features and 3D point cloud; if the scene data includes GPS data, generate latitude, longitude, and altitude information of reference points for geographic coordinate system alignment.

[0146] This step involves parsing the acquired image / video data and using a series of sophisticated and crucial algorithms, including feature extraction, matching, optimization, and alignment, to progressively achieve sparse reconstruction of the target scene. The results of the sparse reconstruction include camera intrinsic data, which details the camera's optical characteristics and is crucial for accurately reconstructing the scene's perspective; camera pose data, precisely describing the camera's position and orientation during data acquisition; 3D point cloud data, visually representing the scene's approximate outline as discrete points; and the correspondence between image features and the 3D point cloud, providing important clues for further accurate reconstruction. It's important to note that when the acquired data includes GPS data, the output pose, point cloud, and other data will be assigned a real-world physical scale. This means that this data accurately reflects the actual size and location of the scene in the real world. At the same time, when the scene data includes GPS data, the latitude, longitude and altitude information (LLA) of the reference point is generated synchronously. This information is like a precise "geographic key", which provides key support for the subsequent accurate alignment of the reconstructed scene into the real-world geographic coordinate system. This allows the reconstructed scene to be seamlessly integrated into the real geographic environment, laying a solid foundation for application scenarios with high geographic accuracy requirements, such as geographic surveying and mapping and urban modeling.

[0147] This embodiment selectively incorporates GPS data during the data acquisition phase and utilizes this GPS data in the sparse reconstruction step to generate camera poses and 3D point clouds with a real-world scale, while simultaneously outputting geographic reference information (such as the latitude, longitude, and altitude of reference points). The technical advantages of this solution are: by deeply integrating GPS data with the visual reconstruction process, it not only solves the problem of scene size distortion caused by scale ambiguity in traditional 3D Gaussian reconstruction methods, ensuring strict consistency between the reconstructed model and the physical scale of the real world; it also achieves seamless and precise alignment between the reconstructed scene and the real-world geographic coordinate system, enabling the output 3D model to naturally possess geospatial attributes. This allows it to be directly integrated as high-precision geographic information data into downstream applications such as GIS systems and digital twin platforms, greatly expanding the practical value and applicability of 3D reconstruction results in fields with stringent requirements for geographic accuracy, such as smart cities, precision surveying and mapping, and land planning.

[0148] It should be noted that, when executing the above embodiment one, visual data needs to be acquired first, and GPS data can be acquired selectively, and the sparse reconstruction operation in step S62 needs to be performed.

[0149] Step S63: Perform data preprocessing based on the 3D point cloud, camera intrinsics, and camera pose output from the sparse reconstruction of the target scene. Data preprocessing may include: performing distortion correction and downsampling on the visual data of the target scene based on the camera intrinsics to obtain a real image; performing target object segmentation and depth estimation on the real image to generate candidate object masks and a depth information map of the target scene; and performing scale restoration processing based on the 3D point cloud, camera pose, candidate object masks, and depth information map to determine the scale alignment parameters.

[0150] It should be noted that the specific implementation of step S63 can refer to step S11 in the above embodiment one, and the second embodiment of the present invention will not be repeated here.

[0151] Step S64: Based on the real image, the 3D point cloud after candidate object mask screening, camera pose, depth information map, and scale alignment parameters, initialize and train the 3D Gaussian model to obtain a coarse-grained 3D Gaussian model of the target scene.

[0152] It should be noted that the specific implementation of step S64 can refer to step S12 in the above embodiment one, and the second embodiment of the present invention will not be repeated here.

[0153] Step S65: Based on the global spatial structure and geometric benchmark represented by the coarse-grained 3D Gaussian model, the target scene is dynamically divided into spatial parts to determine the coordinate range of each sub-block and the associated image of each sub-block.

[0154] The block-segmentation strategy in Embodiment 2 of this invention is a core technical support for efficient and high-precision reconstruction of large-scale scenes. Based on the global spatial structure and geometric benchmark constructed from a coarse model, it scientifically divides the space and filters associated images, solving the pain points of large-scale scenes such as massive data volume, high computational load, and difficulty in global optimization. This lays the foundation for refined modeling and global consistency assurance after block segmentation. Combining the actual spatial range of the reconstruction scene, hardware performance constraints, and reconstruction accuracy requirements, this solution designs a dynamic adaptive block-segmentation strategy. Through configurable reference ranges and data association logic, it achieves automatic matching of the number of blocks and associated data.

[0155] Step S66: Based on the rendered image generated by rendering the target scene using a coarse-grained 3D Gaussian model, a dynamic target mask is generated by combining the real image and the candidate object mask.

[0156] It should be noted that the specific implementation of step S66 can refer to step S13 in the above embodiment one, and the second embodiment of the present invention will not be repeated here.

[0157] Step S67: Based on the associated image corresponding to each sub-block, the part belonging to that sub-block in the coarse-grained 3D Gaussian model, the dynamic target mask, and all input data of the initialization and training of the coarse-grained 3D Gaussian model, perform parallel 3D Gaussian model fine reconstruction and optimization on each sub-block.

[0158] Step S68: Export the parameters of the 3D Gaussian model after the fine reconstruction of each sub-block and its corresponding spatial boundary and geographic coordinate mapping information.

[0159] The difference between Embodiment 2 of this invention and Embodiment 1 lies in the block division step S65 and the corresponding adjustments to subsequent steps S67 and S68. Embodiment 2 uses a technique to dynamically divide the target scene based on the global spatial structure and geometric reference represented by a coarse-grained 3D Gaussian model, and accordingly determines the coordinate range and associated images of each sub-block. This effectively solves the problems of excessive computational load and low reconstruction efficiency caused by the large amount of data and the difficulty of global optimization in large-scale scene reconstruction. This method achieves reasonable allocation and parallel processing of computational resources through dynamic adaptive spatial partitioning, significantly improving the processing efficiency of the reconstruction process. Simultaneously, it employs a two-stage associated image filtering strategy based on camera pose projection and coarse model rendering comparison, ensuring that each sub-block receives sufficient and highly correlated observation data support. This effectively maintains the overall geometric consistency of the scene while ensuring local reconstruction accuracy, laying a solid foundation for subsequent high-quality and high-efficiency block-based refined reconstruction.

[0160] In Embodiment 2 of this invention, the key difference from the block division step in Embodiment 1 is explained in detail. In step S65, based on the global spatial structure and geometric reference represented by the coarse-grained 3D Gaussian model, the target scene is dynamically divided into spatial parts to determine the coordinate range of each sub-block and the associated image of each sub-block. This step can be performed before fine-grained reconstruction and is not sequential with the operation of generating the dynamic target mask in step S66.

[0161] In this embodiment, the xy-plane reference range of each sub-block is first set (the default configuration is 200m×200m, which can be flexibly adjusted according to scene scale and performance requirements); then, the projection coordinates of all camera poses on the xy-plane are extracted, and the minimum bounding rectangle covered by it is calculated as the initial scene range. Based on this, it is expanded by 20% (which can be dynamically adjusted according to the actual shooting situation of the camera) to form the final overall scene range (the purpose of the expansion is to completely cover the effective data of the scene edge and avoid reconstruction blind spots caused by boundary truncation). The number of blocks is dynamically calculated as follows: block_x=ceil(scene_range_x / reference_range_x), block_y=ceil(scene_range_y / reference_range_y), where scene_range_x and scene_range_y are the total xy-plane range of the scene after expansion, and ceil is the rounding function to ensure that the entire scene is completely covered and there are no cases where the block size is insufficient. Subsequently, in order to balance the efficiency and effectiveness of associated image screening and avoid invalid images from consuming computing resources and affecting reconstruction accuracy, this embodiment designs a two-stage strategy of "initial coarse screening + coarse model fine screening" for the associated image screening of each sub-block.

[0162] Specifically, refer to Figure 7 As shown, determining the associated image for each sub-block may include the following steps:

[0163] Step S71: Perform initial coarse screening based on camera pose projection and sub-block enlarged bounding boxes to obtain a candidate image set.

[0164] This initial screening step focuses on camera center projection positioning, combined with block boundary expansion and small block merging strategies to quickly achieve preliminary correlation between "camera and block" while ensuring sufficient observation data for each block. The specific process is as follows:

[0165] (a) Block Boundary Box Generation: Based on the calculated number of blocks, block reference size, etc., calculate the coordinate range of each block, set a configurable scaling factor (default 10%, adjustable as needed), and generate an enlarged bounding box for each sub-block. The purpose of the enlarged bounding box is to cover the observation camera at the edge of the sub-block scene, avoiding the omission of key perspectives due to the camera pose being close to the block boundary or changes in camera viewpoint.

[0166] (b) Camera-block association determination: Extract the projection of the center coordinates of all cameras onto the xy plane, and use the bounding box intersection detection algorithm to determine whether the projection coordinates of each camera fall within the enlarged bounding box of a certain sub-block. Generate a binary association mask. The image corresponding to the camera with a mask value of True is the initial candidate image of that sub-block.

[0167] (c) Small Block Merging Optimization: To avoid problems such as decreased reconstruction accuracy or overfitting due to insufficient number of camera viewpoints in some blocks, an optional adaptive merging step for small blocks is added. The camera number threshold is determined by the maximum value of "total number of cameras in the scene ÷ total number of blocks × 0.7" and 70 to adapt to different scene scales; small blocks to be merged are identified and marked according to the threshold; the adjacent blocks of the block to be merged are retrieved to determine the optimal merge block object, and a new boundary is formed based on the bounding boxes of the two blocks, and the coordinates and block size after merging are updated synchronously. All candidate cameras of the small blocks are associated with the merged blocks.

[0168] The initial screening in this step not only quickly eliminates images that are not spatially related to the sub-blocks (improving screening efficiency by more than 40%), but also ensures that each sub-block has sufficient observation angles by merging small blocks, providing a reasonable and sufficient set of candidate images for subsequent refined screening.

[0169] Step S72: For each image in the candidate image set, render it based on the complete coarse-grained 3D Gaussian model and the model after masking the corresponding Gaussian for the sub-block, and calculate the structural similarity loss between the two rendered images.

[0170] This step uses a coarse-grained 3D Gaussian model as its core, and further filters out associated images that substantially contribute to the reconstruction of this sub-block through projection rendering comparison and adaptive threshold judgment. The specific process is as follows:

[0171] (a) Rendering data preparation: Load the Gaussian parameters of the coarse model, the camera parameters (intrinsic and extrinsic) corresponding to the candidate images, and preset parameters such as background color and content threshold.

[0172] (b) Dual-mode rendering comparison: For each candidate image, two rendering operations are performed: the first is based on the complete coarse model to obtain the original rendering image (img_org_gs); the second is to set the Gaussian transparency within the current sub-block range in the coarse model to 0 (i.e., occlude the Gaussian corresponding to this sub-block) and render the comparison image after masking the sub-block (img_masked_gs).

[0173] (c) SSIM Loss Calculation: Calculate the Structural Similarity (SSIM) between the original rendered image and the comparison image, and obtain the difference loss between the two using the formula "loss = 1.0 - SSIM". The larger the loss value, the higher the proportion and stronger the contribution of the scene content corresponding to this sub-block in the image.

[0174] Through a two-stage screening strategy, the final identified associated images not only ensured effective coverage of the sub-block scene but also eliminated redundant images that did not contribute substantially, providing accurate and efficient observational data support for subsequent block-based fine modeling.

[0175] Step S73: Images with a structural similarity loss greater than a preset content threshold are identified as associated images of the sub-block. This step is an adaptive threshold determination: if the loss value is greater than the preset content_threshold, the image is directly identified as an associated image of the sub-block.

[0176] This embodiment upgrades the image filtering process from coarse spatial location association to refined content contribution evaluation. Specifically, the initial coarse screening based on camera pose projection and sub-block enlarged bounding boxes efficiently eliminates a large number of images that are not directly related to the target sub-block in space, significantly improving screening efficiency and reducing unnecessary computation. Furthermore, the key step of rendering and calculating structural similarity loss based on a complete coarse-grained 3D Gaussian model and a model with the corresponding Gaussian model masked for the sub-block is used to accurately identify perspectives that are actually visible and substantially contribute to the sub-block scene content from the candidate images. This effectively avoids misjudgments caused by camera pose proximity but actual field of view obstruction or the target sub-block having too small a proportion in the image, thus providing more accurate and high-quality observation data input for subsequent refined segmented reconstruction, fundamentally improving the accuracy and reliability of segmented reconstruction.

[0177] In one embodiment, reference is also made to Figure 7 As shown, determining the associated images for each sub-block may further include the following steps: Step S74: If the number of associated images after judgment is less than a preset proportion of the total number of candidate images, the content threshold is lowered and judgment is performed again. For example, if the number of associated images after preliminary judgment is less than 50% of the initial total number of candidate images, the threshold is lowered to content_threshold / 2 for re-filtering.

[0178] This adaptive adjustment mechanism effectively avoids the omission of associated images due to improper initial threshold settings, ensuring that each sub-block has sufficient and necessary data support when performing parallel fine reconstruction. This improves the robustness and adaptability of the overall reconstruction method to different data distribution scenarios while ensuring the accuracy of block data processing.

[0179] In one embodiment, step S67, as the core link between the block-based strategy and global fusion, uses data such as the block-based results, coarse model, and dynamic mask as input to perform parallel fine-tuning on each sub-block. The sub-block tuning generally follows the core training framework of coarse model reconstruction, but performs triple targeted optimizations to meet the needs of detail restoration and resource control.

[0180] First, refine the configuration of hyperparameters. Adjust key hyperparameters, such as the learning rate, number of iterations, and Gaussian gradient threshold, to address the convergence characteristics of local detail modeling and ensure accurate local structure reproduction.

[0181] Second, dynamic interference removal. Based on the dynamic target mask generated in step S66, moving targets are masked during sub-block training to avoid floating artifacts or geometric shifts in the static structure caused by dynamic interference. Specifically, the calculation of the loss function mentioned in step S64 is enhanced with mask processing to ensure that pixels within the masked area do not contribute to the optimization process.

[0182] Third, when performing parallel 3D Gaussian model fine reconstruction and optimization on each sub-block, in order to solve the problems of surge in number, memory explosion and excessive data volume caused by Gaussian splitting and cloning during fine-tuning, and to balance optimization accuracy and resource consumption, a dynamic pruning mechanism is introduced in a fixed iteration cycle, including: First, rendering all related view images of the sub-block and calculating the pixel contribution of each Gaussian element in each view; then, determining the pruning threshold based on the contribution ranking of each Gaussian element and the preset quantile; finally, removing Gaussian elements with a contribution lower than the pruning threshold.

[0183] In this embodiment, all associated views of the current sub-block are rendered, and the pixel contribution of each Gaussian in each view is calculated. After sorting each Gaussian in descending order of contribution, the average of the top n contributions is calculated as the final contribution of that Gaussian. Based on the contribution distribution of all Gaussians in the sub-block, the x% quantile is taken as the pruning threshold (to ensure that the core contributing Gaussians are retained). Redundant Gaussians with a contribution below the threshold are directly removed, which not only ensures detail coverage but also reduces memory usage.

[0184] This embodiment introduces a dynamic pruning mechanism during block-based parallel fine-grained reconstruction, effectively addressing the issues of memory overload and computational efficiency degradation caused by a surge in the number of Gaussian elements, while ensuring accurate restoration of local details. The scheme sorts and thresholds Gaussian elements based on their actual pixel contribution across different viewpoints, prioritizing the removal of redundant Gaussian elements that contribute minimally to the final rendering result. This significantly reduces the computational and storage overhead of the model while maintaining high-precision details, thereby improving resource utilization efficiency and overall robustness in large-scale scene reconstruction.

[0185] Step S68 above, which exports scene reconstruction block data, serves as the final output of scene reconstruction and is a crucial bridge connecting the reconstruction process with downstream applications. Its core objective is to output standardized, reusable block reconstruction data, providing precise geospatial support for downstream applications. Specifically, it is implemented as follows: Based on the preset spatial range of each sub-block, the effective Gaussian data after refined reconstruction is precisely filtered and trimmed, retaining only the core Gaussian parameters within the sub-block boundaries (including xyz, scales, rotations, shs_dc, shs_rest, opacities, etc.). Then, following the standard 3D Gaussian model data structure, the Gaussian results for each sub-block are exported as a PLY format file, ensuring good data compatibility and readability. Simultaneously, the corresponding metadata file is exported, containing key information such as the unique block ID, sub-block spatial boundary coordinates, the mapping relationship between the local and global coordinate systems, and the geographic reference datum (e.g., GPS coordinate system parameters, projection method). This step utilizes the coordinate transformation rules preset in the metadata, allowing downstream applications to quickly transform sub-block data from a local coordinate system to a real geographic coordinate system based on any reference point. This provides standardized and high-precision data support for downstream needs such as GIS system integration, digital twin scenario construction, and spatial analysis.

[0186] Based on the same inventive concept, Embodiment 2 of the present invention also provides a 3D reconstruction device based on 3D Gaussian, referring to... Figure 8 As shown, the device may include: an acquisition module 81, a sparse reconstruction module 82, a data preprocessing module 51, a coarse model reconstruction module 52, a block segmentation module 83, a dynamic mask generation module 53, a sub-block parallel fine reconstruction module 84, and an export module 85. Its working principle is as follows:

[0187] The acquisition module 81 is used to acquire scene data of the target scene, the scene data including visual data and optionally including GPS data; the visual data includes image data and / or video data;

[0188] The sparse reconstruction module 82 is used to perform sparse reconstruction based on the scene data, generate camera intrinsic parameters, camera pose, 3D point cloud, and the correspondence between image features and 3D point cloud; and generate latitude, longitude and height information of reference points for geographic coordinate system alignment when the scene data includes GPS data.

[0189] The data preprocessing module 51 is used to perform data preprocessing based on the 3D point cloud, camera intrinsic parameters, and camera pose output from the sparse reconstruction of the target scene. The data preprocessing includes: performing distortion correction and downsampling processing on the visual data of the target scene based on the camera intrinsic parameters to obtain a real image; performing target object segmentation and depth estimation processing on the real image to generate candidate object masks and depth information maps of the target scene; and performing scale restoration processing based on the 3D point cloud, camera pose, candidate object masks, and depth information maps to determine scale alignment parameters.

[0190] The coarse model reconstruction module 52 is used to initialize and train a 3D Gaussian model based on real images, 3D point clouds after candidate object mask screening, camera pose, depth information map, and scale alignment parameters to obtain a coarse-grained 3D Gaussian model of the target scene.

[0191] The segmentation module 83 is used to dynamically divide the target scene into spaces based on the global spatial structure and geometric benchmark represented by the coarse-grained 3D Gaussian model, so as to determine the coordinate range of each sub-block and the associated image of each sub-block.

[0192] The dynamic mask generation module 53 is used to generate a dynamic target mask based on the rendered image generated by rendering the target scene using a coarse-grained 3D Gaussian model, combined with the real image and the candidate object mask.

[0193] The fine-grained reconstruction module 54 is used to perform fine-grained reconstruction and optimization of the coarse-grained 3D Gaussian model based on all input data that the coarse-grained 3D Gaussian model has been initialized and trained, including the coarse-grained 3D Gaussian model, dynamic target mask, and coarse-grained 3D Gaussian model.

[0194] The sub-block parallel fine-reconstruction module 84 is used to perform parallel 3D Gaussian model fine-reconstruction and optimization on each sub-block based on the associated image corresponding to each sub-block, the part of the coarse-grained 3D Gaussian model belonging to the sub-block, the dynamic target mask, and all input data of the initialization and training of the coarse-grained 3D Gaussian model.

[0195] The export module 85 is used to export the parameters of the 3D Gaussian model after the fine reconstruction of each sub-block and its corresponding spatial boundary and geographic coordinate mapping information.

[0196] In one embodiment, the segmentation module 83 is specifically used for:

[0197] An initial coarse screening is performed based on camera pose projection and sub-block enlarged bounding boxes to obtain a candidate image set;

[0198] For each image in the candidate image set, render it based on the complete coarse-grained 3D Gaussian model and the model after masking the Gaussian corresponding to the sub-block, and calculate the structural similarity loss between the two rendered images.

[0199] Images with a structural similarity loss greater than a preset content threshold are identified as associated images of the sub-block.

[0200] In one embodiment, the block segmentation module 83 can also be used to: if the number of associated images after determination is less than a preset proportion of the total number of candidate images, then lower the content threshold and re-determine.

[0201] In one embodiment, the sub-block parallel fine-grained reconstruction module 84 is specifically used for:

[0202] Render all associated view images of the sub-block and calculate the pixel contribution of each Gaussian element in each view.

[0203] The pruning threshold is determined based on the contribution ranking of each Gaussian element and the preset quantile.

[0204] Remove Gaussian units whose contribution is below the pruning threshold.

[0205] Based on the same inventive concept, this embodiment of the invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described 3D Gaussian-based three-dimensional reconstruction method.

[0206] Based on the same inventive concept, this embodiment of the invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described 3D Gaussian-based three-dimensional reconstruction method.

[0207] The principles by which the above-mentioned devices, media, and related equipment in the embodiments of the present invention solve the problem are similar to those of the aforementioned methods. Therefore, their implementation can refer to the implementation of the aforementioned methods, and repeated details will not be repeated.

[0208] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0209] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0210] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0211] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0212] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of equivalents of this invention, this invention is also intended to include these modifications and variations.

Claims

1. A 3D reconstruction method based on 3D Gaussian, characterized in that, include: Data preprocessing is performed based on the 3D point cloud, camera intrinsics, and camera pose output from the sparse reconstruction of the target scene. The data preprocessing includes: performing distortion correction and downsampling on the visual data of the target scene based on the camera intrinsic parameters to obtain a real image; performing target object segmentation and depth estimation on the real image to generate a candidate object mask and a depth information map of the target scene; and performing scale restoration processing based on the 3D point cloud, the camera pose, the candidate object mask, and the depth information map to determine scale alignment parameters. Based on the real image, the 3D point cloud after candidate object masking, the camera pose, the depth information map, and the scale alignment parameters, a 3D Gaussian model is initialized and trained to obtain a coarse-grained 3D Gaussian model of the target scene. Based on the rendered image generated by rendering the target scene using the coarse-grained 3D Gaussian model, a dynamic target mask is generated by combining the real image and the candidate object mask. Based on the coarse-grained 3D Gaussian model, the dynamic target mask, and all input data initialized and trained by the coarse-grained 3D Gaussian model, the coarse-grained 3D Gaussian model is refined and optimized.

2. The method according to claim 1, characterized in that, Before performing fine reconstruction and optimization on the coarse-grained 3D Gaussian model, the following steps are also included: Based on the global spatial structure and geometric benchmark represented by the coarse-grained 3D Gaussian model, the target scene is dynamically divided into spatial parts to determine the coordinate range of each sub-block and the associated image of each sub-block. Accordingly, based on the coarse-grained 3D Gaussian model, the dynamic target mask, and all input data initialized and trained by the coarse-grained 3D Gaussian model, the coarse-grained 3D Gaussian model is refined through reconstruction and optimization, including: Based on the associated images corresponding to each sub-block, the part of the coarse-grained 3D Gaussian model belonging to that sub-block, the dynamic target mask, and all input data of the initialization and training of the coarse-grained 3D Gaussian model, parallel 3D Gaussian model fine reconstruction and optimization are performed on each sub-block. Export the parameters of the 3D Gaussian model after the fine reconstruction of each sub-block, as well as the corresponding spatial boundary and geographic coordinate mapping information.

3. The method according to claim 2, characterized in that, Determining the associated image for each sub-block includes: An initial coarse screening is performed based on camera pose projection and sub-block enlarged bounding boxes to obtain a candidate image set; For each image in the candidate image set, render it based on the complete coarse-grained 3D Gaussian model and the model after masking the Gaussian corresponding to the sub-block, and calculate the structural similarity loss between the two rendered images. Images with a structural similarity loss greater than a preset content threshold are identified as associated images of the sub-block.

4. The method according to claim 2, characterized in that, The parallel 3D Gaussian model refinement and optimization of each sub-block includes: Render all associated view images of the sub-block and calculate the pixel contribution of each Gaussian element in each view. The pruning threshold is determined based on the contribution ranking of each Gaussian element and the preset quantile. Remove Gaussian units whose contribution is below the pruning threshold.

5. The method according to claim 1, characterized in that, Scale restoration processing is performed based on the 3D point cloud, the camera pose, the candidate object mask, and the depth information map to determine scale alignment parameters, including: Based on the correspondence between the candidate object mask and the 3D point cloud and image features, valid 3D points not covered by the candidate object mask are selected. Based on the camera pose, the effective 3D points are transformed to the camera coordinate system and their inverse depth is calculated; Based on the coordinates of the two-dimensional feature points in the image corresponding to the effective three-dimensional points, the corresponding inverse depth values ​​are extracted from the depth information map. Based on the inverse depth set of the effective 3D points and the inverse depth value set extracted from the depth information map, a scale factor and an offset factor are calculated, and the scale factor and the offset factor are used as the scale alignment parameters.

6. The method according to claim 1, characterized in that, Based on the real image, the 3D point cloud after candidate object masking, the camera pose, the depth information map, and the scale alignment parameters, a 3D Gaussian model is initialized and trained to obtain a coarse-grained 3D Gaussian model of the target scene, including: The 3D Gaussian model is initialized based on the selected 3D point cloud and camera pose, and the scale parameter of each Gaussian element perpendicular to the normal direction of the scene surface is constrained to zero to form a planar Gaussian representation. The parameters of each Gaussian element include: 3D spatial coordinates, scale parameter, rotation parameter, spherical harmonic function coefficient, and transparency parameter. During training, the current 3D Gaussian model is rendered based on the real image, the depth information map, and the scale alignment parameters to obtain the rendered image and the rendered depth map. Calculate a composite loss function; wherein the composite loss function includes a color loss term, a depth regularization loss term, and a normal consistency loss term; the color loss term is a weighted loss between the rendered image and the corresponding real image, including an L1 loss component and a structural similarity loss component; the depth regularization loss term is a weighted loss between the rendered depth map and the scaled depth map obtained based on the depth information map and the scale alignment parameter, including an L1 loss component and a structural similarity loss component; the normal consistency loss term is used to constrain the consistency between the rendered surface normal and the geometric normal represented by the coarse-grained 3D Gaussian model; The parameters of the 3D Gaussian model are updated using a dynamic gradient optimization strategy; wherein the gradient weight of each Gaussian element is determined based on its projected area, and the gradient scaling factor is determined based on the ratio of its depth to the average depth of the scene. Repeat the above training steps, composite loss function calculation steps, and update steps until the preset training stopping condition is met to obtain the coarse-grained 3D Gaussian model.

7. The method according to claim 1, characterized in that, Based on the rendered image generated from the target scene using the coarse-grained 3D Gaussian model, and combined with the real image and the candidate object mask, a dynamic target mask is generated, including: Calculate the RGB difference matrix and structural similarity matrix between the rendered image and the real image; The RGB difference matrix and structural similarity matrix are normalized and weighted and fused to obtain a pixel-level combination loss heatmap; Based on the candidate object mask, each target instance region is identified, and the average loss value of the combined loss heatmap within each instance region is calculated. An adaptive threshold is set based on the distribution of the average loss value of all instance regions. Instance regions with an average loss value higher than the adaptive threshold are identified as dynamic targets, and the dynamic target mask is generated.

8. The method according to any one of claims 1 to 7, characterized in that, Before performing sparse reconstruction of the scene data of the target scene, the following steps are also included: Acquire scene data of the target scene, the scene data including visual data and optionally including GPS data; the visual data includes image data and / or video data; Based on the scene data, sparse reconstruction is performed to generate camera intrinsic parameters, camera pose, 3D point cloud, and the correspondence between image features and 3D point cloud; with the scene data including GPS data, latitude, longitude, and altitude information of reference points for geographic coordinate system alignment is generated.

9. A three-dimensional reconstruction device based on 3D Gaussian, characterized in that, include: The data preprocessing module is used to preprocess data based on the 3D point cloud, camera intrinsic parameters, and camera pose output from the sparse reconstruction of the target scene. The data preprocessing includes: performing distortion correction and downsampling on the visual data of the target scene based on the camera intrinsic parameters to obtain a real image; performing target object segmentation and depth estimation on the real image to generate a candidate object mask and a depth information map of the target scene; and performing scale restoration processing based on the 3D point cloud, the camera pose, the candidate object mask, and the depth information map to determine scale alignment parameters. The coarse model reconstruction module is used to initialize and train a 3D Gaussian model based on the real image, the 3D point cloud after the candidate object mask is filtered, the camera pose, the depth information map, and the scale alignment parameters, so as to obtain a coarse-grained 3D Gaussian model of the target scene. The dynamic mask generation module is used to generate a dynamic target mask based on the rendered image generated by rendering the target scene using the coarse-grained 3D Gaussian model, combined with the real image and the candidate object mask. The fine-grained reconstruction module is used to perform fine-grained reconstruction and optimization of the coarse-grained 3D Gaussian model based on the coarse-grained 3D Gaussian model, the dynamic target mask, and all input data initialized and trained by the coarse-grained 3D Gaussian model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the 3D Gaussian-based three-dimensional reconstruction method as described in any one of claims 1 to 8.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the 3D reconstruction method based on 3D Gaussian as described in any one of claims 1 to 8.

Images