A method and system for three-dimensional reconstruction of cross-source data based on improved Gaussian sputtering

Abstract

The application discloses a kind of three-dimensional reconstruction methods and systems of cross-source data based on improved Gaussian sputtering, comprising: collecting the unmanned aerial vehicle oblique image and ground panoramic image of target area, construct space-time correlation dataset;Based on multi-view geometric constraint, establish space-time coding matching point pair through adaptive feature pyramid, carry out intelligent incremental cross-source data sparse reconstruction, output point cloud and camera parameter;Improved Gaussian sputtering is used, three-dimensional Gaussian kernel is compressed into two-dimensional Gaussian primitive by double tangent vector constraint and is fitted with surface geometry, multi-scale reconstruction is realized;The primitive parameter is optimized using differentiable renderer, and multi-scale fine reconstruction is completed by gradient back propagation, to generate high-precision three-dimensional model.The application provides multi-scale accurate geometric prior input and accurate camera pose for three-dimensional reconstruction, greatly reduces the dependence on professional manual operation in traditional three-dimensional reconstruction method, and significantly improves the geometric accuracy and visual fidelity of reconstruction result.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, specifically to a cross-source data 3D reconstruction method and system based on improved Gaussian sputtering. Background Technology

[0002] The goal of image-based 3D scene reconstruction is to convert a series of photographs or videos of the scene to be reconstructed into a digital 3D model that can be computed, analyzed, and manipulated. This challenging and enduring problem is fundamental to machines' understanding of the complexity of the real world. 3D reconstruction technology is crucial for a wide range of applications, including robot navigation, medical imaging and diagnosis, digital twin preservation of historical sites, augmented / virtual reality, and autonomous driving.

[0003] The development of 3D reconstruction technology has progressed from the foundational geometric theories of the 1960s, to the maturity of multi-view geometric methods in the early 21st century, and then to the revolutionary breakthroughs in deep learning and differentiable rendering in recent years. Traditional geometric methods are based on the framework of Structure from Motion (SfM) and Multi-View Stereo (MVS) algorithms, with representative systems such as COLMAP. Their performance heavily relies on manually designed feature extraction algorithms, performs poorly in weakly textured regions and scenes with repetitive textures, and has low efficiency in dense reconstruction.

[0004] With the improvement of computing power and the development of deep learning technology, machine learning-based 3D reconstruction methods have gradually emerged. Neural Radiation Field (NeRF) has achieved unprecedented rendering quality by implicitly representing the scene, redefining the technical route of scene representation and rendering. However, its computational overhead is large, especially in application scenarios that are very sensitive to latency. 3D Gaussian Sputtering (3DGS) combines the advantages of explicit representation and differentiable rendering. It uses a learnable 3D Gaussian kernel to represent the scene and uses a highly parallel processing flow. It significantly improves the rendering speed while maintaining high-quality reconstruction, becoming a new paradigm in the field of 3D reconstruction. However, its explicit representation characteristics can lead to problems such as outlier Gaussians affecting the reconstruction effect and the difficulty of the 3D Gaussian kernel in determining the reconstructed surface.

[0005] Therefore, how to solve the problems of insufficient geometric accuracy and outlier Gaussian artifacts in image-based 3D Gaussian sputtering is a technical problem that researchers in this field urgently need to solve. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for cross-source data 3D reconstruction based on improved Gaussian sputtering, so as to solve the problems mentioned in the background art.

[0007] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0008] A method and system for cross-source data 3D reconstruction based on improved Gaussian sputtering includes the following steps:

[0009] Acquire UAV oblique photography sequences and ground panoramic image sequences for reconstructing the target area, and construct a spatiotemporally correlated cross-source target scene dataset;

[0010] Cross-source data joint sparse reconstruction is based on the multi-view geometric constraints of the constructed cross-source target scene dataset. Cross-source matching point pairs with spatiotemporal coding are established through an adaptive scale feature pyramid. Incremental sparse reconstruction is performed based on spatiotemporal coding, and sparse 3D point cloud and camera extrinsic matrix are output.

[0011] An improved Gaussian sputtering explicit radiation field is used to compress the original three-dimensional Gaussian kernel into two-dimensional Gaussian primitives through double tangent vector constraints. The primitives are forced to fit the reconstructed surface geometric manifold, and multi-scale reconstruction is performed on the sparse reconstruction results of cross-source data.

[0012] Based on a differentiable renderer, geometric regularization optimization is performed on 2D Gaussian primitives. Primitive parameters are adjusted through gradient backpropagation to achieve progressive reconstruction from sparse point clouds to dense geometry. Multi-scale reconstruction is performed on the sparse reconstruction results of cross-source data to generate a high-precision 3D model corresponding to the target scene.

[0013] Furthermore, the acquisition of UAV oblique photography sequences and ground panoramic image sequences for reconstructing the target area, and the construction of a spatiotemporally correlated target scene dataset, includes:

[0014] Using a drone equipped with a camera sensor, the target area to be reconstructed is photographed from multiple angles with time-series encoding according to the planned path, and the drone oblique photography sequence is obtained.

[0015] Using a panoramic camera, capture ground panoramic video of the reconstructed target area, supplement cross-source perspective and target scene detail information, and obtain ground panoramic image sequence;

[0016] Viewpoint unfolding and resolution are performed on the ground panoramic image sequence. A dynamic equal-angle segmentation strategy is adopted. Centered on the panoramic acquisition point, the video frames are geometrically projected from a sphere to a plane to generate a distortion-free image sequence covering 360° horizontally, with azimuth angles evenly divided. , N is a natural number.

[0017] Furthermore, the step of establishing cross-source matching point pairs with spatiotemporal coding through an adaptive scale feature pyramid includes:

[0018] Construct a multimodal feature pyramid, and establish adaptive scale feature pyramids for UAV tilted image sequences and ground panoramic unfolded image sequences respectively, and attach spatiotemporal coding vectors to feature points. ,in, For time-series encoding, UAV imagery is encoded in flight strip order, while ground imagery is encoded in video frame time sequence. Spherical coordinate encoding is obtained by back-projecting pixel coordinates onto the initial coordinate system of the acquisition device;

[0019] Scale-invariant feature transform is used to extract cross-source feature points, and temporal adjacent frame matching is used to perform feature matching on cross-source image sequences with spatiotemporal coding to obtain cross-source matching point pairs with spatiotemporal coding.

[0020] Furthermore, the incremental sparse reconstruction based on spatiotemporal coding, outputting a sparse 3D point cloud and camera extrinsic matrix, includes:

[0021] Based on cross-source matching point pairs with spatiotemporal coding, a nonlinear optimization problem under multi-view geometric constraints is constructed. Reliable matching pairs are selected through geometric verification, and high-matching image pairs are selected for initial reconstruction. The essential matrix is ​​decomposed to obtain the initial camera pose and triangulated to generate seed point clouds.

[0022] Incremental registration is performed on image sequences of cross-source target scene datasets. New camera poses are calculated and point clouds are expanded using the perspective n-point method. This process is repeated based on the spatiotemporal encoding of cross-source matching point pairs.

[0023] When the number of incremental point clouds or registered images exceeds a threshold, the point cloud and camera extrinsic matrix are optimized using bundle adjustment until all images are registered, and the final sparse point cloud and camera pose are output.

[0024] Furthermore, the original three-dimensional Gaussian kernel is compressed into a two-dimensional Gaussian element through bitangent vector constraints, including:

[0025] Using a point in three-dimensional space And a set of mutually perpendicular unit tangent vectors and Let represent the compressed two-dimensional Gaussian element, whose normal vector is defined as ;

[0026] Use scaling factor and The variance of the two-dimensional Gaussian plane is controlled by the following vectors, which represent the variances along the tangent vector. and The scaling factor of the direction is used to achieve a dimensionality reduction representation from a three-dimensional Gaussian kernel to a two-dimensional Gaussian element while preserving the surface geometric features;

[0027] Ultimately, the two-dimensional Gaussian element starts from the center point. Covariance Matrix Full definition The superscript T indicates transpose. Define the spatial orientation of the two-dimensional Gaussian elements as rotation matrices. For the scaling matrix, define the shape of the two-dimensional Gaussian element, where This represents a diagonal matrix with diagonal elements of 1.

[0028] Furthermore, the forced primitive fitting reconstruction of the surface geometric manifold includes:

[0029] The sparse point cloud is used to initialize the two-dimensional Gaussian primitives and determine the center point. At the feature points of the target geometry to be reconstructed, the reconstructed surface is initially fitted;

[0030] The K-nearest neighbor method is used to search for the nearest neighbor points. Principal component analysis is performed on the local point cloud to calculate the covariance matrix of the neighborhood points. Eigenvalue decomposition is then performed on the covariance matrix to obtain the eigenvalues. and eigenvectors ;

[0031] Based on local geometry, the minimum eigenvalue corresponds to the eigenvector. Let be the plane normal vector of the two-dimensional Gaussian element, and the bitangent vector be initialized as follows: Scaling factor and This allows the two-dimensional Gaussian elements to be close to the reconstructed surface during initialization.

[0032] Furthermore, the geometric regularization optimization of the two-dimensional Gaussian elements based on the differentiable renderer includes:

[0033] Using deep regularization loss As the loss function for deep optimization, where, For the first The weighting of the points where a ray intersects the plane. For the first The weighting of the points where a ray intersects the plane. For the first The depth value from a two-dimensional Gaussian element to the imaging plane. For the first The depth values ​​from a 2D Gaussian element to the imaging plane are minimized by minimizing the depth difference between intersection points.

[0034] Using normal consistency loss As the loss function for normal optimization, where For a two-dimensional Gaussian element, the plane normal vector is... The normal vector is estimated from the depth map; by aligning the normal vectors of the two-dimensional Gaussian distribution with the normal vectors of the actual surface, the reconstructed surface is ensured to be smooth and the local geometry is accurate.

[0035] Furthermore, the adjustment of primitive parameters through gradient backpropagation includes:

[0036] Using an end-to-end differentiable rendering pipeline, the position coordinates of 2D Gaussian primitives, the direction of bitangent vectors, scaling factors, and color and transparency features are all included in the set of trainable variables. During each iteration of rendering, a composite view is generated through differentiable rasterization, and pixel-level loss is calculated with the real observed image.

[0037] An adaptive momentum optimizer is used to backpropagate the error signal along the direction of the rendering loss gradient. For geometric parameters, the coordinates of the primitive center point and the direction of the tangent vector are optimized to improve geometric accuracy. For appearance parameters, the primitive opacity and color function are optimized to improve rendering realism.

[0038] Based on the loss gradient threshold, 2D Gaussian elements are split to fit complex geometries, and anomalous 2D Gaussian elements are eliminated based on opacity and tangential scaling.

[0039] Furthermore, the step of performing multi-scale reconstruction on the sparse reconstruction results of cross-source data to generate a high-precision model corresponding to the target scene includes:

[0040] Based on spatiotemporal coding, the overall target scene is reconstructed from coarse to fine across cross-source data at multiple scales. The wide-area coverage characteristics of UAV oblique photogrammetry data are used to construct the overall geometric framework and structural features of the target scene.

[0041] By using spatiotemporal coding, high-resolution detail information from ground panoramic images is gradually fused together, and fine geometric features of wall textures and decorative components are added at a fine-grained level, forming a hierarchical reconstruction process.

[0042] High-performance GPUs are used to achieve efficient modeling and optimization, ultimately generating a high-precision model corresponding to the target scene.

[0043] This invention also provides a cross-source data 3D reconstruction system based on improved Gaussian sputtering, used to implement the cross-source data 3D reconstruction method based on improved Gaussian sputtering as described above, comprising:

[0044] Data acquisition unit: used to acquire UAV oblique photography sequences and ground panoramic image sequences to reconstruct the target area, build a spatiotemporally correlated target scene dataset, and provide multi-view geometric constraints;

[0045] Spatiotemporal coding unit: used to perform data preprocessing on the target scene dataset, including scale matching of cross-source image sequences, viewpoint unfolding solution of panoramic image sequences, and extraction of cross-source feature points to obtain cross-source matching point pairs with spatiotemporal coding;

[0046] Sparse Reconstruction Unit: Used for intelligent incremental registration of image sequences in cross-source target scene datasets, incremental sparse reconstruction based on spatiotemporal coding, and outputting sparse 3D point cloud and camera extrinsic matrix as the initial input of the 3D reconstruction unit.

[0047] The 3D reconstruction unit is used to employ an improved Gaussian sputtering explicit radiation field, introduce bitangent vector constraints and geometric regularization optimization, and then perform multi-scale reconstruction of the target reconstruction scene based on machine learning using a differentiable renderer to generate a high-precision model corresponding to the target scene, thus completing the 3D reconstruction task.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] This invention uses cross-source data joint sparse reconstruction. Based on the multi-view geometric constraints of the spatiotemporally correlated target scene dataset, it establishes cross-source matching point pairs with spatiotemporal encoding through an adaptive scale feature pyramid. Incremental sparse reconstruction is then performed based on the spatiotemporal encoding, providing accurate geometric prior inputs at multiple scales and precise camera poses for 3D reconstruction.

[0050] This invention introduces bitangent vector constraints and geometric regularization optimization to compress the original three-dimensional Gaussian kernel into two-dimensional Gaussian primitives, forcing the primitives to fit the reconstructed surface geometric manifold, thereby obtaining more accurate geometric reconstruction results and greatly reducing the problem of reconstruction artifacts caused by outlier Gaussians in the original three-dimensional Gaussian kernel.

[0051] This invention transforms the traditional manual model reconstruction process into end-to-end high-fidelity 3D reconstruction using machine learning methods by employing an end-to-end differentiable rendering pipeline and an adaptive momentum optimizer. This allows the model to automatically learn the optimal reconstruction strategy, directly acquire the geometric and visual information of the target reconstruction scene from cross-source dataset images, and automatically perform 3D reconstruction. This significantly reduces the reliance on professional manual operation in traditional 3D reconstruction methods, while significantly improving the geometric accuracy and visual fidelity of the reconstruction results. Attached Figure Description

[0052] Figure 1 This is a flowchart of a cross-source data 3D reconstruction method based on improved Gaussian sputtering according to the present invention.

[0053] Figure 2 This is a flowchart illustrating a specific embodiment of a cross-source data 3D reconstruction method based on improved Gaussian sputtering according to the present invention.

[0054] Figure 3 This is a schematic diagram of a cross-source data 3D reconstruction system framework based on improved Gaussian sputtering according to the present invention. Detailed Implementation

[0055] To make the technical solutions and advantages of the present invention clearer, the specific embodiments will be described in detail below with reference to the accompanying drawings. It should be particularly noted that these embodiments are only used to illustrate the core ideas of the present invention and are not intended to limit the invention. Technical solutions obtained by those skilled in the art through simple adjustments or equivalent substitutions without departing from the principles of the present invention should all be considered to fall within the protection scope of the present invention.

[0056] like Figure 1 As shown, in order to achieve the above objectives, this invention proposes a cross-source data 3D reconstruction method based on improved Gaussian sputtering, comprising the following steps:

[0057] Step 1: Obtain the UAV oblique photography sequence and ground panoramic image sequence for reconstructing the target area, and construct a spatiotemporally correlated target scene dataset, which specifically includes the following steps:

[0058] Step 1.1: Using a drone equipped with a camera sensor, perform time-series encoded multi-angle oblique photography of the reconstructed target area according to the planned path to obtain the drone oblique photography sequence.

[0059] Step 1.2: Use a panoramic camera to capture ground panoramic video of the reconstructed target area, supplement cross-source perspective and target scene detail information, and obtain ground panoramic image sequence; Figure 2 Figure (a) illustrates the data acquisition method in a specific embodiment and the resulting target scene dataset sequence.

[0060] Step 1.3 involves viewpoint unfolding and solving the ground panoramic image sequence. A dynamic equal-angle segmentation strategy is adopted, with the panoramic acquisition point as the center. The video frames are geometrically projected from a sphere to a plane to generate a distortion-free image sequence covering 360° horizontally. The dynamic segmentation angle is determined by the scene complexity, and the azimuth angle is equally divided into intervals. , N is a natural number.

[0061] Among them, the dynamic equal-angle segmentation strategy is a well-known technology. It adopts a projection method with fixed angle intervals to divide the panoramic video frame into N parts according to the horizontal azimuth angle (N is a preset constant), and each part corresponds to a virtual pinhole camera view.

[0062] Step 1.4: Integrate the UAV oblique photography sequence with the distortion-free panoramic image sequence after viewpoint expansion to construct a spatiotemporally correlated cross-source target scene dataset.

[0063] Step 2 involves joint sparse reconstruction of cross-source data. Based on the multi-view geometric constraints of the constructed cross-source target scene dataset, cross-source matching point pairs with spatiotemporal encoding are established through an adaptive scale feature pyramid. Incremental sparse reconstruction is then performed based on the spatiotemporal encoding, outputting a sparse 3D point cloud and camera extrinsic matrix. The adaptive scale feature pyramid is a well-known technique in the field of 3D reconstruction developed to simulate human visual "gazing" behavior. Its core idea is to allow the network to adaptively and dynamically select or fuse the most suitable feature scale based on image content (especially geometric information), rather than treating all regions uniformly. Step 2 specifically includes:

[0064] Step 2.1: Construct a multimodal feature pyramid. Establish adaptive scale feature pyramids for both the UAV tilted image sequence and the ground panoramic unfolded image sequence, and attach spatiotemporal coding vectors to the feature points. ,in, For time-series encoding, UAV imagery is encoded in flight strip order, while ground imagery is encoded in video frame time sequence. Spherical coordinate encoding is obtained by back-projecting pixel coordinates onto the initial coordinate system of the acquisition device.

[0065] Among them, multimodal feature pyramids are a well-known 3D reconstruction technique that integrates multi-scale information from different sensors to address challenges such as texture loss, illumination variations, and geometric blurring from a single data source. Multi-scale feature pyramids are a known technique that extracts and fuses features at different resolution levels of the same image, thereby simultaneously capturing rich details and high-level semantic information to improve the accuracy and robustness of 3D reconstruction.

[0066] Step 2.2: Use scale-invariant feature transform to extract cross-source feature points, and use temporal adjacent frame matching to perform feature matching on cross-source image sequences with spatiotemporal coding to obtain cross-source matching point pairs with spatiotemporal coding.

[0067] Among them, Scale Invariant Feature Transform (SIFT) is a traditional algorithm used to extract keypoint features from images that are invariant to changes in scale, rotation, and illumination, aiming to provide stable and reliable image matching points for 3D reconstruction. Temporal adjacent frame matching is a well-known technique that estimates camera motion pose and scene structure by analyzing the feature points shared between consecutive frames (video sequences) that are temporally continuous with the current image.

[0068] Step 2.3: Based on cross-source matching point pairs with spatiotemporal coding, a nonlinear optimization problem under multi-view geometric constraints is constructed. Reliable matching pairs are screened through geometric verification, and high-matching image pairs are selected for initial reconstruction. The essential matrix is ​​decomposed to obtain the initial camera pose and triangulated to generate seed point clouds. Triangulation to generate seed point clouds is a geometric calculation method that generates initial sparse 3D point clouds from two-dimensional image points that have been successfully matched from multiple different views by calculating the optimal intersection point of their lines of sight in three-dimensional space. This is a well-known technology.

[0069] Step 2.4: Incrementally register the image sequences of the cross-source target scene dataset, calculate the new camera pose and expand the point cloud using the perspective n-point method, and repeat this process based on the spatiotemporal encoding of the cross-source matching point pairs.

[0070] Incremental registration is a well-known technique that involves aligning and fusing newly acquired 3D scan data (or images) with existing global scene models in a step-by-step and sequential manner to gradually expand and refine the entire 3D reconstruction result.

[0071] Step 2.5: When the number of incremental point clouds or registered images exceeds a preset threshold, the point cloud and camera extrinsic matrix are optimized using bundle adjustment until all images are registered. The final sparse point cloud and camera pose are then output. The sparse reconstruction result is as follows: Figure 2 As shown in Figure (b), which illustrates the sparse reconstructed feature point cloud and the pose of cross-source images in three-dimensional space according to a specific embodiment, providing geometric priors for further dense reconstruction.

[0072] Bundle Adjustment (BA) is a well-known technique that optimizes all camera parameters and 3D point coordinates simultaneously to minimize the reprojection error when projected onto a 2D image, thereby obtaining the globally optimal 3D structure and motion parameters.

[0073] Step 3 involves employing an improved Gaussian sputtering explicit radiation field. By using bitangent vector constraints, the original 3D Gaussian kernel is compressed into 2D Gaussian primitives, forcing the primitives to conform to the reconstructed surface geometry. Multi-scale reconstruction is then performed on the sparse reconstruction results from the cross-source dataset, specifically including:

[0074] Step 3.1, using a point in three-dimensional space And a set of mutually perpendicular unit tangent vectors and Let represent the compressed two-dimensional Gaussian element, whose normal vector is defined as .

[0075] Step 3.2, using scaling factor and The variance of the two-dimensional Gaussian plane is controlled by the following vectors, which represent the variances along the tangent vector. and The scaling factor of the direction is used to achieve a dimensionality reduction representation from a three-dimensional Gaussian kernel to a two-dimensional Gaussian element while preserving the surface geometry.

[0076] Step 3.3, the final two-dimensional Gaussian element starts from the center point. Covariance Matrix Fully defined ,in For the rotation matrix, define the spatial orientation of the Gaussian elements. Define the shape of the Gaussian element for the scaling matrix. This represents a diagonal matrix with diagonal elements of 1.

[0077] Step 3.4: Based on the sparse point cloud obtained in Step 2.5, initialize the two-dimensional Gaussian primitives and determine the center point. The reconstructed surface is initially fitted at the feature points of the target geometry.

[0078] Step 3.5: Use the K-nearest neighbor method to search for the nearest neighbor point, perform principal component analysis on the local point cloud, calculate the covariance matrix of the neighborhood points, and perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues. and eigenvectors There is a one-to-one correspondence between eigenvalues ​​and eigenvectors. The size of the eigenvalue corresponds to the length of the eigenvector. The eigenvector corresponding to the eigenvalue with the smallest eigenvalue is selected as the plane normal vector.

[0079] K-Nearest Neighbors (KNN) is a well-known algorithm used to quickly find the nearest neighbors of each point in a point cloud based on spatial distance, supporting outlier filtering, surface reconstruction, and feature description.

[0080] Step 3.6: Based on local geometry, the minimum eigenvalue corresponds to the eigenvector. Let be the normal vector of the two-dimensional Gaussian plane, and initialize the bitangent vectors as follows: Scaling factor and This allows the two-dimensional Gaussian elements to be close to the reconstructed surface during initialization.

[0081] Specifically, based on the sparse reconstruction of the point cloud, two-dimensional Gaussian primitives are initialized so that they are closely attached to the surface of the target building being reconstructed; for example... Figure 2As shown in (c), the geometric prior information contained in the sparse point cloud generated by motion recovery structure is fully utilized. By calculating the eigenvector of the covariance matrix of the local neighborhood of the point cloud, the direction of the principal tangent plane of each primitive is determined. This transforms the isotropic spherical Gaussian kernel in traditional 3DGS into a two-dimensional planar primitive with a clear direction, thereby obtaining a more accurate geometric reconstruction result and greatly reducing the problem of reconstruction artifacts caused by outlier Gaussians in the original three-dimensional Gaussian kernel.

[0082] Step 4: Perform geometric regularization optimization on the 2D Gaussian primitives based on the differentiable renderer, adjust the primitive parameters through gradient backpropagation, and achieve progressive reconstruction from sparse point cloud to dense geometry to generate a high-precision 3D model corresponding to the target scene. Specifically, this includes:

[0083] Step 4.1, use depth regularization loss As the loss function for deep optimization, For the first The weighting of the points where a ray intersects the plane. For the first The depth value from a two-dimensional Gaussian element to the imaging plane. For the first The depth values ​​from two-dimensional Gaussian elements to the imaging plane are calculated; by minimizing the depth difference between intersection points, depth blur caused by dispersion is avoided, Gaussian artifacts are reduced, and the reconstructed geometry is made clearer and more accurate.

[0084] Step 4.2, use normal consistency loss As the loss function for normal optimization, where For a two-dimensional Gaussian element, the plane normal vector is... The normal vector is estimated from the depth map; by aligning the normal vectors of the two-dimensional Gaussian distribution with the normal vectors of the actual surface, the reconstructed surface is ensured to be smooth and the local geometry is accurate.

[0085] Step 4.3 uses an end-to-end differentiable rendering pipeline to include all parameters such as the position coordinates of the two-dimensional Gaussian primitives, the direction of the bitangent vector, the scaling factor, and the color and transparency features into the set of trainable variables. During each iteration of rendering, a composite view is generated through differentiable rasterization, and pixel-level loss is calculated with the real observed image.

[0086] Differentiable rasterization is a rendering mechanism in the original Gaussian sputtering algorithm. This mechanism projects three-dimensional Gaussian primitives onto a two-dimensional screen space and performs gradient backpropagation. This mechanism makes the gradient at every step of the entire rendering process computable, thereby allowing the optimization of the parameters of the three-dimensional model itself through direct backpropagation of the final image loss. This is a well-known technique.

[0087] Step 4.4: An adaptive momentum optimizer is used to backpropagate the error signal along the direction of the rendering loss gradient. For geometric parameters, the coordinates of the primitive center point and the direction of the tangent vector are optimized to improve geometric accuracy. For appearance parameters, the primitive opacity and color function are optimized to improve rendering realism.

[0088] Among them, the Adaptive Momentum Optimizer (Adam) is a deep learning optimization algorithm that combines momentum method and adaptive learning rate mechanism. It is used to efficiently optimize the large-scale nonlinear parameters of neural networks in 3D reconstruction to achieve fast and stable convergence, and is a well-known technology.

[0089] Step 4.5: Based on the loss gradient threshold, split the Gaussian elements to fit the complex geometry, and remove abnormal Gaussian elements based on the opacity and tangential scaling ratio to achieve high-precision modeling optimization.

[0090] Step 4.6: Based on spatiotemporal coding, perform multi-scale reconstruction of the overall target scene from coarse to fine cross-source data, and utilize the wide-area coverage characteristics of UAV oblique photogrammetry data to construct the overall geometric framework and main structural features of the scene.

[0091] Step 4.7 involves gradually fusing high-resolution detail information from the ground panoramic image through spatiotemporal coding, supplementing fine geometric features such as wall textures and decorative components at a fine-grained level, and forming a hierarchical reconstruction process.

[0092] Step 4.8 utilizes a high-performance GPU to achieve efficient modeling and optimization, ultimately generating a high-precision model file corresponding to the target scene, such as... Figure 2 As shown in (d).

[0093] like Figure 3 As shown, the present invention also provides a cross-source data 3D reconstruction system based on improved Gaussian sputtering, used to implement the above-mentioned cross-source data 3D reconstruction method based on improved Gaussian sputtering, including a data acquisition unit 201, a spatiotemporal coding unit 202, a sparse reconstruction unit 203 and a 3D reconstruction unit 204, wherein the data acquisition unit 201, the spatiotemporal coding unit 202, the sparse reconstruction unit 203 and the 3D reconstruction unit 204 are all computer programs.

[0094] Data acquisition unit 201: used to acquire UAV oblique photography sequences and ground panoramic image sequences for reconstructing the target area, construct a spatiotemporally correlated target scene dataset, and provide multi-view geometric constraints.

[0095] Spatiotemporal coding unit 202: is used to perform data preprocessing on the above target scene dataset, including scale matching of cross-source image sequences, viewpoint unfolding solution of panoramic image sequences, and extraction of cross-source feature points, to obtain cross-source matching point pairs with spatiotemporal coding.

[0096] Sparse Reconstruction Unit 203: Used for intelligent incremental registration of image sequences of cross-source target scene datasets, incremental sparse reconstruction based on spatiotemporal coding, outputting sparse 3D point clouds and camera extrinsic matrix, providing accurate geometric priors for subsequent 3D reconstruction.

[0097] 3D Reconstruction Unit 204: Used to employ an improved Gaussian sputtering explicit radiation field, introduce bitangent vector constraints and geometric regularization optimization, and then perform multi-scale reconstruction of the target reconstruction scene based on machine learning using a differentiable renderer to generate a high-precision model corresponding to the target scene, thereby completing the 3D reconstruction task.

[0098] In the specific implementation process, the data acquisition unit 201 uses a drone system equipped with a camera sensor to carry out multi-angle tilt shooting of the target reconstruction area according to the preset flight path. During the shooting process, a time sequence coding mark is added to each image, and finally a drone tilt photography sequence with time sequence information is obtained. A panoramic camera device is used to carry out ground panoramic video acquisition of the target reconstruction area. Through multi-angle continuous shooting, complete scene panoramic image data is obtained, thereby supplementing cross-source information from different perspectives and capturing richer scene detail features to form a complete ground panoramic image sequence.

[0099] Then, the spatiotemporal coding unit 202 preprocesses the acquired data, performs viewpoint unfolding processing on the acquired panoramic video data, and converts the original video frames into a horizontally complete and distortion-free planar image sequence with uniformly distributed viewpoint intervals for each image. Subsequently, multi-scale feature pyramids are established for the UAV tilted image sequence and the ground panoramic unfolded image sequence, respectively. Scale-invariant feature transformation is used to extract cross-source feature points, and spatiotemporal coding vectors are added to the feature points. Then, cross-source matching of feature points is performed to obtain cross-source matching point pairs with spatiotemporal coding.

[0100] Subsequently, the sparse reconstruction unit 203 gradually restores the three-dimensional structure of the scene using an incremental reconstruction method based on the spatiotemporal coding information in the preprocessed dataset. This unit generates sparse point cloud data through iterative calculation and optimizes the camera extrinsic parameters, providing accurate geometric constraints and initial conditions for subsequent fine three-dimensional reconstruction.

[0101] Finally, the 3D reconstruction unit 204 adopts an improved Gaussian sputtering explicit radiation field, based on a differentiable rendering pipeline, and uses machine learning methods to perform learnable reconstruction of the target scene. It utilizes a high-performance GPU to achieve efficient modeling and optimization, and finally generates a high-precision model file corresponding to the target scene. After export, it can be applied to tasks such as robot navigation, medical imaging and diagnosis, digital twin protection of historical sites, augmented / virtual reality, and autonomous driving.

[0102] The present invention has provided a detailed description of a cross-source data 3D reconstruction method and system based on improved Gaussian sputtering. Specific embodiments have been used to illustrate the invention, but are only for the purpose of helping to understand the method and its core ideas. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, and are not intended to limit its scope of protection. Those skilled in the art should understand that any equivalent transformations or modifications made to the embodiments without departing from the core concept of the present invention should be included within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for three-dimensional reconstruction of cross-source data based on improved Gaussian sputtering, characterized in that, The method comprises the following steps: obtaining a UAV oblique photography sequence and a ground panoramic image sequence of a target region to be reconstructed, and constructing a cross-source target scene dataset with spatiotemporal correlation; cross-source data joint sparse reconstruction, based on multi-view geometric constraints of the constructed cross-source target scene dataset, cross-source matching point pairs with spatiotemporal coding are established through an adaptive scale feature pyramid, incremental sparse reconstruction is performed based on the spatiotemporal coding, and sparse three-dimensional point clouds and camera extrinsic parameter matrices are outputted; multi-scale reconstruction is performed on the sparse reconstruction result of the cross-source data by using an improved Gaussian sputtering explicit radiation field, and the original three-dimensional Gaussian kernel is compressed into a two-dimensional Gaussian primitive through double tangent vector constraints, and the primitive is forced to fit the reconstruction surface geometric manifold; the original three-dimensional Gaussian kernel is compressed into a two-dimensional Gaussian primitive through double tangent vector constraints, including: A two-dimensional Gaussian cell after compression is represented by a point in three-dimensional space and a set of mutually orthogonal unit tangent vectors and with the normal vector defined as ; Using scaling factors And Control the variance of the two-dimensional Gaussian plane, respectively, representing the scaling factor along the tangent vector And The direction, so as to realize the dimension reduction representation of three-dimensional Gaussian kernel to two-dimensional Gaussian element while preserving the surface geometric features; The final two-dimensional Gaussian cell is defined by a center point and a covariance matrix completely where the superscript T denotes the transpose, is a rotation matrix defining the spatial orientation of the two-dimensional Gaussian cell is a scaling matrix defining the shape of the two-dimensional Gaussian cell, where denotes a diagonal matrix with diagonal elements based on the differentiable renderer, geometric regularization optimization is performed on the two-dimensional Gaussian primitive, the primitive parameters are adjusted through gradient back propagation, and incremental reconstruction from sparse point clouds to dense geometry is realized to perform multi-scale reconstruction on the sparse reconstruction result of the cross-source data, and a high-precision three-dimensional model corresponding to the target scene is generated.

2. The method of claim 1, wherein the method is based on improved Gaussian spatter for cross-source data 3D reconstruction. The method comprises the following steps: using a UAV equipped with a camera sensor to perform time-encoding multi-angle oblique photography on the target region to be reconstructed according to a planned path, and obtaining a UAV oblique photography sequence; using a panoramic camera to shoot a ground panoramic video of the target region to be reconstructed, and supplementing cross-source view and target scene detail information, and obtaining a ground panoramic image sequence; The view point unfolding calculation is performed on the ground panoramic image sequence, a dynamic equiangular segmentation strategy is adopted, the video frames are projected through the spherical to planar geometry with the panoramic collection point as the center, a non-distortion image sequence covering horizontal 360° is generated, and the azimuth angle is equally divided , , N is a natural number.

3. The method of claim 1, wherein the method is based on improved Gaussian spatter for cross-source data 3D reconstruction. The method comprises the following steps: A multi-modal feature pyramid is constructed, and adaptive scale feature pyramids are established for the UAV oblique image sequence and the ground panoramic unwinding image sequence respectively, and a space-time coding vector is added to the feature points wherein, is a time sequence coding, the UAV image is coded according to the flight strip sequence, and the ground image is coded according to the video frame time sequence; is a spherical coordinate coding, which is obtained by inversely projecting the pixel coordinates to the initial coordinate system of the collecting device. using scale-invariant feature transformation to extract cross-source feature points, and using time-sequentially adjacent frame matching method to perform feature matching on the cross-source image sequence with spatiotemporal coding, and obtaining cross-source matching point pairs with spatiotemporal coding.

4. The method of claim 1, wherein the method is based on improved Gaussian sputtering for cross-source data 3D reconstruction. The method comprises the following steps: based on the cross-source matching point pairs with spatiotemporal coding, a nonlinear optimization problem under multi-view geometric constraints is constructed, reliable matching pairs are selected through geometric verification, high-matching-degree image pairs are selected for initialization reconstruction, an essential matrix is decomposed to obtain initial camera poses and seed point clouds are generated through triangulation; the image sequence of the cross-source target scene dataset is registered incrementally, new camera poses are calculated through the perspective-n-point method and point clouds are expanded, and the process is cycled according to the spatiotemporal coding of the cross-source matching point pairs; when the number of incremental point clouds or registered images is greater than a threshold, the point clouds and camera extrinsic parameter matrices are optimized through bundle adjustment, until all images are registered, and final sparse point clouds and camera poses are outputted.

5. The method of claim 1, wherein the method is based on improved Gaussian sputtering for cross-source data 3D reconstruction. The method comprises the following steps: using the sparse point cloud, initializing a two-dimensional Gaussian cell, determining a center point performing a preliminary fitting of the reconstructed surface at a feature point of the reconstructed target geometry The K nearest neighbor method is used to search the nearest points, principal component analysis is performed on the local point cloud, a covariance matrix of the neighborhood points is calculated, eigenvalue decomposition is performed on the covariance matrix, and eigenvalues and eigenvectors are obtained Based on local geometry, the minimum eigenvalue corresponds to the eigenvector For the plane normal vector of the two-dimensional Gaussian cell, the bitangent vector is initialized as , the scaling coefficient and , so that the two-dimensional Gaussian cell is close to the reconstructed surface at initialization.

6. The method of claim 1, wherein the method is based on improved Gaussian spatter for cross-source data 3D reconstruction. The method comprises the following steps: Using a depth regularization loss as a depth optimization loss function, wherein, is a blending weight for the intersection of the th ray with the plane, is a blending weight for the intersection of the th ray with the plane, is a depth value for the th two-dimensional Gaussian cell to the imaging plane, is a depth value for the th two-dimensional Gaussian cell to the imaging plane; by minimizing the depth disparity between intersections; Using normal consistency loss As a normal optimization loss function, where is the plane normal of a two-dimensional Gaussian cell, is the normal vector estimated by the depth map; by aligning the normal of the two-dimensional Gaussian distribution and the normal of the actual surface, it ensures that the reconstructed surface is smooth and the local geometry is accurate.

7. The method of claim 1, wherein the method is based on improved Gaussian sputtering for cross-source data 3D reconstruction. The method comprises the following steps: The method comprises the following steps: The position coordinates, bivector directions, scaling coefficients, color and transparency features of the two-dimensional Gaussian primitives are all included in the trainable variable set using an end-to-end differentiable rendering pipeline, and the synthesized view is generated by differentiable rasterization at each iteration rendering, and the pixel-level loss is calculated with the real observation image; An adaptive momentum optimizer is used to propagate error signals in the direction of the rendering loss gradient, and for geometric parameters, the primitive center point coordinates and bivector directions are optimized to improve geometric accuracy; for appearance parameters, the primitive opacity and color functions are optimized to improve rendering realism; According to the loss gradient threshold, the two-dimensional Gaussian primitives are split to fit complex geometric structures, and abnormal two-dimensional Gaussian primitives are removed according to the opacity and bivector scaling ratio.

8. The method of claim 1, wherein the method is based on improved Gaussian sputtering for cross-source data 3D reconstruction. The multi-scale reconstruction of the sparse reconstruction result of the cross-source data is performed to generate a high-precision model corresponding to the target scene, comprising: Based on the spatiotemporal coding, the overall target scene is reconstructed by cross-source data multi-scale reconstruction from coarse to fine, and the wide-area coverage characteristics of the unmanned aerial vehicle oblique photography data are used to construct the overall geometric framework and structural features of the target scene; Through spatiotemporal coding, the high-resolution detailed information of the ground panoramic image is gradually fused to supplement the fine geometric features of the wall texture and decorative components at a fine-grained level, forming a hierarchical reconstruction process; High-performance GPUs are used to realize efficient modeling and optimization, and finally generate a high-precision model corresponding to the target scene.

9. A system for improved Gaussian-spraying cross-source data 3D reconstruction, for implementing the method of any one of claims 1-8, characterized in that, Comprising: A data acquisition unit is used to acquire an unmanned aerial vehicle oblique photography sequence and a ground panoramic image sequence of a reconstruction target area, construct a spatiotemporal associated target scene data set, and provide multi-view geometric constraints; A spatiotemporal coding unit is used for data preprocessing of the target scene data set, including scale matching of cross-source image sequences, viewpoint unfolding calculation of panoramic image sequences, and extraction of cross-source feature points, to obtain cross-source matching point pairs with spatiotemporal coding; A sparse reconstruction unit is used for intelligent incremental registration of image sequences of the cross-source target scene data set, and based on spatiotemporal coding, incremental sparse reconstruction is performed to output sparse three-dimensional point cloud and camera extrinsic parameter matrix as the initial input of the three-dimensional reconstruction unit; A three-dimensional reconstruction unit is used to use an improved Gaussian sputtering explicit radiation field, introduce bivector constraint and geometric regularization optimization, and then based on a differentiable renderer, a multi-scale reconstruction of the target reconstruction scene based on machine learning is performed to generate a high-precision model corresponding to the target scene, completing the three-dimensional reconstruction task.

Images