Pose-unconstrained and prior-free generalizable three-dimensional gaussian sputtering system and method

Abstract

The application discloses a pose-constraint-free and prior-free generalizable three-dimensional Gaussian splatting system and method, belongs to the field of computer vision and three-dimensional reconstruction, and comprises the following steps: inputting multi-view images without pose labeling, extracting multi-scale feature maps, weighting and fusing different scale features to obtain fused feature maps; estimating camera pose parameters by using the fused feature maps; initializing depth by using the multi-view images without pose labeling and the fused feature maps, and outputting an initial depth map; obtaining a high-precision refined depth map by using a pyramid refining structure and an edge perception attention mechanism; and performing high-quality three-dimensional reconstruction by using an explicit 3D Gaussian splatting technology. The application significantly improves the generalization ability, enhances the stability of camera pose estimation, and reduces the pose estimation error by 40%, and has high practicability; under the conditions of no pose and no 3D prior, the error of the three-dimensional reconstruction result is less than or equal to 0.1 mm, the texture restoration degree is greater than or equal to 95%, and the reconstruction precision is improved by more than 30%.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D reconstruction technology, specifically relating to a generalizable 3D Gaussian sputtering system and method without pose constraints and prior knowledge. Background Technology

[0002] With the development of computer vision technology, 3D reconstruction is increasingly widely used in fields such as virtual reality, industrial inspection, and cultural heritage protection. 3D Gaussian sputtering technology, due to its efficient rendering and detail preservation capabilities, has become a research hotspot in the field of 3D reconstruction in recent years. However, in practical applications, 3D Gaussian sputtering technology often faces scenarios where multi-view images lack pose annotations and there is no pre-defined 3D prior information. In such cases, the model needs to achieve generalized reconstruction without fine-tuning for specific scenarios.

[0003] In existing technologies, 3D reconstruction methods are mainly divided into reconstruction schemes that rely on pose information and reconstruction schemes that do not require pose input. Reconstruction schemes that rely on pose information require obtaining camera pose data in advance or manually annotating the pose, which limits their applicability. In reconstruction schemes that do not require pose input, some methods achieve reconstruction by fusing pose estimation and 3D representation techniques, but they suffer from high dependence on ground truth data or pre-trained geometric information and weak generalization ability. The closest prior art to this invention is a method for optimizing 3D Gaussian sputtering without pose input disclosed in Chinese patent CN119006678A, which includes the following steps: First, input a multi-view image without pose annotation, and use a ray prediction model to predict the ray beam distribution in the image and extract distribution features; second, calculate the camera pose parameters corresponding to each view based on the distribution features; then, obtain the initial spatial distribution of the 3D Gaussian point cloud through ray volume density sampling, and calculate the visible shell by combining the view frustum projection and object masking, and construct a training scene based on the initial Gaussian point cloud and the visible shell; finally, introduce a training regularization term containing camera pose parameters during the training process, and achieve 3D reconstruction through loss function optimization. This scheme calculates camera pose by obtaining ray beam distribution features through a ray prediction model and performs scene training by combining a visual shell and Gaussian point cloud initialization. Its core relies mainly on ray beam distribution features for pose estimation and requires obtaining an initial Gaussian distribution through a visual shell and density sampling. It does not involve the integration of explicit 3D representation and self-supervised techniques, nor does it have a specific design for the matching accuracy of pose estimation and the refinement of depth information. The reconstruction accuracy and stability are greatly affected by the initialization process.

[0004] In summary, the existing technology has the following main drawbacks: 1. Insufficient generalization ability: Most existing technologies rely on implicit geometric information or indirect truth correlations in specific scenarios, lack generalization design under no 3D prior conditions, and have not solved the generalization problem under no 3D prior conditions. When facing unseen scenarios, fine-tuning is required to ensure the reconstruction effect, which cannot meet the general reconstruction needs.

[0005] 2. Limited accuracy of camera pose estimation: Existing technologies deduce camera pose through indirect features such as light beam distribution, without considering the feature matching correlation between multiple views. This makes camera pose estimation susceptible to image noise and viewpoint differences, resulting in insufficient stability.

[0006] 3. Coarse depth information: Existing technologies do not have a dedicated refinement mechanism for depth information and only obtain the initial depth distribution through density sampling, resulting in a lack of detail in the reconstruction results and making it difficult to meet the requirements for high-precision reconstruction.

[0007] 4. Reliance on 3D prior information and reduced reconstruction quality: Although some existing technologies do not rely on pose input, they still need to rely on 3D prior information such as pre-trained geometric models and scene templates. In unfamiliar scenes without any prior information, the reconstruction quality is greatly reduced. Summary of the Invention

[0008] To address the problems of weak generalization ability, limited camera pose estimation accuracy, coarse depth information, reliance on 3D prior information, and degraded reconstruction quality in existing poseless 3D reconstruction methods, this invention provides a generalizable 3D Gaussian sputtering system and method without pose constraints and prior information.

[0009] This invention aims to provide a high-precision and highly stable generalizable 3D reconstruction method that requires no pose annotation, no 3D prior information, no scene fine-tuning, and solves the 3D reconstruction quality problem in unfamiliar scenes. It is applicable to high-precision 3D reconstruction scenarios that do not require pose annotation, no 3D prior information, and do not rely on fine-tuning.

[0010] The technical solution adopted by this invention to solve the technical problem is as follows: This invention provides a generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge, comprising the following steps: Step S1: Input a multi-view image without pose annotation, extract multi-scale feature maps, perform weighted fusion of features at different scales, and obtain a fused feature map; Step S2: Estimate camera pose parameters using the fused feature map; Step S3: Initialize the depth using the pose-unannotated multi-view image and the fused feature map, and output the initial depth map; Step S4: Use a pyramid-shaped refinement structure and an edge-aware attention mechanism to obtain a high-precision refined depth map; Step S5: Perform high-quality 3D reconstruction using explicit 3D Gaussian sputtering technology.

[0011] Furthermore, in step S1, an improved ResNet-50 network is first used to perform layer-by-layer convolution and pooling operations on each input image to extract multi-scale feature maps. Then, the features at different scales are weighted and fused to obtain a fused feature map of the same dimension.

[0012] Furthermore, in step S2, a cross-correlation attention mechanism is first used to calculate the pixel-level matching similarity matrix between any two fused feature maps, and then matching point pairs with a confidence score ≥ 0.8 are selected to construct a multi-view matching map. A convolutional neural network and a fully connected network are used to process the matching map to obtain the rotation matrix and translation vector of each matching map relative to the reference view. The pose transformation consistency error between any three fused feature maps is calculated. If the error is greater than the threshold, the confidence threshold is adjusted, and matching point pairs are re-selected until the consistency error is less than or equal to the threshold. Finally, the camera pose parameters that have passed the consistency verification are output.

[0013] Furthermore, in step S3, using pose-unlabeled multi-view images and fused feature maps, a photometric consistency loss function between views is constructed based on the principle of self-supervised multi-view stereo matching; the depth prediction network is optimized through unsupervised training, and the initial depth map of each image is output through the optimized depth prediction network.

[0014] Furthermore, the view-to-view photometric consistency loss function is: , For the i-th multi-view image without pose annotation, For the j-th multi-view image without pose annotation, To obtain the pixel-level matching similarity matrix from feature map i to fused feature map j, This is the initial depth map.

[0015] Furthermore, in step S4, the initial depth map is decomposed into four scales through a pyramid-shaped refinement structure, and depth correction is performed on each scale, gradually optimizing from coarse to fine.

[0016] Furthermore, in step S4, edge information of the image is extracted from the fused feature map through the edge-aware attention mechanism, and the depth value of the edge region is given a higher optimization weight; the optimization objective is set to minimize the depth consistency loss.

[0017] Furthermore, the specific implementation process of step S5 is as follows: Based on the high-precision refined depth map and camera pose parameters, an initial Gaussian point cloud is generated in 3D space. The fused feature map is mapped to 3D space, and feature attributes are assigned to each Gaussian point to construct an explicit 3D feature field. The initial Gaussian point cloud is projected onto each view plane through a Gaussian kernel function. The photometric loss between the rendered image and the high-precision refined depth map obtained in step S4 is calculated. Combined with the sparsity loss of the initial Gaussian point cloud, the Gaussian point parameters are iteratively optimized. The 3D reconstruction results are output, including a 3D point cloud model and a mesh model.

[0018] Furthermore, it also includes step S6: training and inference, the specific implementation process of which is as follows: (1) Training phase: Training dataset: A general multi-view dataset without pose annotation and without fixed scene; Loss function: Total loss ; For luminance loss, For pose return loss, The core loss function is depth-dependent. For rendering loss, This is a sparsity loss; Training parameters: The optimizer is AdamW, the batch size is 8, the training epochs are 200, and the learning rate decay strategy is used; (2) Reasoning stage: The system can directly input multi-view images of any scene without pose annotations, without the need for fine-tuning or additional parameter adjustments. The system's linear pipeline processing outputs 3D reconstruction results.

[0019] This invention provides a generalizable 3D Gaussian sputtering system without attitude constraints or prior knowledge, comprising: The multi-view feature extraction module is used to extract multi-scale feature maps, perform weighted fusion of features at different scales, and obtain a fused feature map with a unified dimension. A matching-aware pose estimation network is used to estimate camera pose parameters using fused feature maps; The self-supervised depth initialization module is used to initialize the depth using pose-unlabeled multi-view images and fused feature maps, and output an initial depth map. The depth refinement module is used to obtain high-precision refined depth maps by employing a pyramid-style refinement structure and an edge-aware attention mechanism. The explicit Gaussian sputtering reconstruction module is used to implement Gaussian initialization, explicit 3D representation fusion, and Gaussian sputtering and optimization to obtain 3D reconstruction results.

[0020] The beneficial effects of this invention are: 1. Significantly improved generalization ability: To address the problem of insufficient generalization ability in existing technologies, this invention, through self-supervised training and general feature extraction design, can be directly applied to the 3D reconstruction of unfamiliar scenes without any 3D prior information or scene fine-tuning. The reconstruction accuracy in unseen scenes is improved by more than 30% compared with existing technologies.

[0021] 2. Enhanced stability of camera pose estimation: To address the problem of limited accuracy in existing camera pose estimation technologies, this invention utilizes a matching-aware pose estimation network to effectively reduce the impact of noise and viewpoint differences through feature matching and consistency verification. The pose estimation error is reduced by 40% compared to existing technologies, and the consistency error is stabilized within 0.05 in complex scenes.

[0022] 3. Depth information accuracy optimization: To address the problem of coarse depth information in existing technologies, this invention utilizes the pyramid-shaped refinement structure of the depth refinement module and the edge-aware attention mechanism to improve the detail representation of the depth map and reduce the depth error in the edge region by 50%, providing support for high-precision 3D reconstruction.

[0023] 4. Reconstruction quality meets standards: In response to the problem that existing technologies rely on 3D prior information and suffer from reduced reconstruction quality, this invention achieves a 3D reconstruction result error of ≤0.1mm and a texture restoration accuracy of ≥95% under conditions of no pose and no 3D prior information. This is more than 30% higher than the reconstruction accuracy of existing technologies and can meet high-end needs such as industrial inspection and high-precision modeling.

[0024] 5. High practicality: The reasoning process of this invention does not require manual intervention and does not rely on professional equipment to mark poses, which lowers the application threshold of 3D reconstruction and is suitable for emergency reconstruction, unknown scene detection and other scenarios where there is a lack of 3D prior information. Attached Figure Description

[0025] Figure 1 The diagram illustrates the implementation principle of a generalizable 3D Gaussian sputtering system without attitude constraints or prior knowledge, as provided by this invention.

[0026] Figure 2 This is a schematic diagram illustrating the implementation principle of a matching-sensory pose estimation network. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings.

[0028] In a first aspect, the present invention provides a generalizable three-dimensional Gaussian sputtering system without attitude constraints and prior knowledge.

[0029] This invention provides a generalizable 3D Gaussian sputtering system without pose constraints or prior knowledge. It achieves prior-free generalized reconstruction primarily by fusing explicit 3D representation, self-supervised techniques, and dedicated network modules. Figure 1As shown, its overall architecture mainly includes five core modules: a multi-view feature extraction module, a matching-aware pose estimation network, a self-supervised depth initialization module, a depth refinement module, and an explicit Gaussian sputtering reconstruction module. These modules are connected in series and form a feedback loop. Specifically, the output of the multi-view feature extraction module (fused feature map) is fed into the matching-aware pose estimation network and the self-supervised depth initialization module, respectively. The matching-aware pose estimation network outputs camera pose parameters, and the self-supervised depth initialization module outputs an initial depth map. The camera pose parameters and the initial depth map are input into the depth refinement module. The output of the depth refinement module (high-precision refined depth map) is combined with the fused feature map and input into the explicit Gaussian sputtering reconstruction module to complete high-quality 3D reconstruction.

[0030] This invention provides a pose-constrained and prior-free generalizable 3D Gaussian sputtering system. Taking pose-unannotated multi-view images as input, it processes the data through a linear pipeline of five core modules: First, a multi-view feature extraction module extracts multi-view features and outputs a fused feature map. Next, a matching-aware pose estimation network and a self-supervised depth initialization module jointly estimate camera pose parameters and initialize depth. Then, a depth refinement module refines the depth map using a pyramid strategy, outputting a high-precision refined depth map. Finally, an explicit Gaussian sputtering reconstruction module performs high-quality 3D reconstruction using explicit 3D Gaussian sputtering technology. The key to this linear pipeline processing is the self-supervised learning achieved through an internal consistency verification mechanism, ultimately outputting a 3D point cloud model and mesh model with millimeter-level precision, achieving high-quality 3D reconstruction without relying on any 3D prior information.

[0031] According to this invention, the multi-view feature extraction module can specifically employ an improved ResNet-50 network, removing the last three fully connected layers and adding two-level feature fusion layers at the end of the network. The improved ResNet-50 network performs layer-by-layer convolution and pooling operations on the input multi-view image without pose annotations to extract multi-scale feature maps. The feature fusion layers then perform weighted fusion of features at different scales to obtain a fused feature map of uniform dimension.

[0032] According to this invention, the matching-aware pose estimation network adopts a "top-bottom" design, such as... Figure 2As shown, it mainly includes: the network main module, feature matching branch, pose regression branch, and consistency verification branch. The top network module receives the fused feature maps, while the three bottom branches process them sequentially: ① The feature matching branch uses a cross-correlation attention mechanism to calculate the pixel-level matching similarity matrix between any two fused feature maps, filters matching point pairs with a confidence score ≥ 0.8, and constructs a multi-view matching map; ② The pose regression branch takes the matching map as input and regresses 6-DOF pose parameters (rotation matrix R and translation vector t) from the matching map through a 6-layer convolutional neural network (CNN) and a 3-layer fully connected network, outputting the rotation matrix (3×3 dimension) and translation vector (3×1 dimension) of each matching map relative to the reference view; ③ The consistency verification branch performs geometric verification of the multi-view pose, calculates the pose transformation consistency error between any three fused feature maps, and if the error > the threshold (0.05), it feeds back to adjust the confidence threshold of the feature matching branch, re-filters matching point pairs, until the consistency error ≤ the threshold (0.05), forming an optimization loop to ensure the accuracy and consistency of the output pose; finally, it outputs the camera pose parameters after consistency verification: rotation matrix and translation vector.

[0033] Preferably, the convolutional kernel size of the convolutional neural network is 3×3, the stride is 1, and the padding is 1.

[0034] According to the present invention, the self-supervised depth initialization module constructs a photometric consistency loss function between views based on the principle of self-supervised multi-view stereo matching; the depth prediction network is optimized through unsupervised training, without the need for ground truth depth data, and the optimized depth prediction network can directly output the initial depth map of each image.

[0035] According to this invention, the depth refinement module specifically employs a pyramid-shaped refinement structure and an edge-aware attention mechanism to obtain a high-precision refined depth map. The pyramid-shaped refinement structure achieves a four-level pyramid-shaped optimization process from coarse to fine depth, while the edge-aware attention mechanism extracts image edge information from the fused feature map, assigning higher optimization weights to the depth values ​​of edge regions to reduce edge blurring.

[0036] According to the present invention, the explicit Gaussian sputtering reconstruction module is used to realize Gaussian initialization, explicit 3D representation fusion, and Gaussian sputtering and optimization to obtain three-dimensional reconstruction results, specifically including a three-dimensional point cloud model and a mesh model, with an accuracy of up to 0.1 mm.

[0037] In a second aspect, the present invention provides a generalizable three-dimensional Gaussian sputtering method that is free from attitude constraints and prior knowledge, implemented by a generalizable three-dimensional Gaussian sputtering system that is free from attitude constraints and prior knowledge, as described in the first aspect.

[0038] This invention provides a generalizable 3D Gaussian sputtering method without attitude constraints or prior knowledge, which mainly includes the following steps: Step S1: Multi-view feature extraction; Inputting multi-view images without pose annotations, an improved ResNet-50 network is used to perform layer-by-layer convolution and pooling operations on each image to extract multi-scale feature maps. The features at different scales are then weighted and fused through a feature fusion layer to obtain a fused feature map with a uniform dimension of 256×H / 8×W / 8 (H and W are the height and width of the image, respectively), providing data support for subsequent pose estimation and depth initialization.

[0039] The input multi-view images without pose annotations have a resolution range of 640×480~2048×1536 and a quantity of ≥3.

[0040] The extracted multi-scale feature maps mainly include shallow texture features, mid-level semantic features, and deep geometric features.

[0041] Step S2: Estimate camera pose parameters; The fused feature map obtained in step S1 is input, and a pixel-level matching similarity matrix between any two fused feature maps is calculated using a cross-correlation attention mechanism. Matching point pairs with a confidence score ≥ 0.8 are selected to construct a multi-view matching map. Using the matching map as input, a rotation matrix (3×3 dimension) and translation vector (3×1 dimension) of each matching map relative to the reference view are output through a 6-layer convolutional neural network (CNN) and a 3-layer fully connected network. The pose transformation consistency error between any three fused feature maps is calculated. If the error is greater than the threshold (0.05), the confidence threshold is adjusted, and matching point pairs are re-selected until the consistency error is less than or equal to the threshold (0.05). Finally, the camera pose parameters that have passed the consistency verification are output: rotation matrix and translation vector.

[0042] Step S3: Initialize depth; Input multi-view images without pose annotations and the fused feature map obtained in step S1. Based on the principle of self-supervised multi-view stereo matching, construct the photometric consistency loss function between views. Optimize the depth prediction network through unsupervised training. No ground truth depth data is required. The optimized depth prediction network can directly output the initial depth map of each image. The depth range is adaptively adjusted according to the image scene with an accuracy of 0.01m.

[0043] The constructed view-to-view photometric consistency loss function is: ,in, For the i-th multi-view image without pose annotation, For the j-th multi-view image without pose annotation, To obtain the pixel-level matching similarity matrix from feature map i to fused feature map j, This is the initial depth map.

[0044] Step S4: Refine the depth map using a pyramid strategy; The fused feature map obtained in step S1, the camera pose parameters obtained in step S2, and the initial depth map obtained in step S3 are input. A high-precision refined depth map is obtained using a pyramid-style thinning structure and an edge-aware attention mechanism. Specifically, the initial depth map is decomposed into four scales (1×1, 2×2, 4×4, 8×8) using a pyramid-style thinning structure, and depth correction is performed on each scale, gradually optimizing from coarse to fine. The edge-aware attention mechanism extracts image edge information from the fused feature map obtained in step S1, assigning higher optimization weights to the depth values ​​of edge regions to reduce edge blurring. The optimization objective is set to minimize the depth consistency loss.

[0045] The mathematical expression for the deep consistency loss is: ,in, To refine the depth map with high precision. Refine the depth map for other views. This is a pixel-level similarity matrix.

[0046] Specifically, a four-level pyramid-style optimization process, from coarse to fine, is achieved through a pyramid-style refinement structure. The specific implementation process is as follows: The process employs four progressively refined scales: global depth correction is performed at an 8×8 coarse scale (error controlled within 0.5mm); regional gradient optimization is performed at a 4×4 medium scale (error controlled within 0.2mm); an edge-aware mechanism is introduced at a 2×2 fine scale to enhance contour region accuracy (error controlled within 0.1mm); and finally, pixel-level calibration is performed at a 1×1 ultra-fine scale (error controlled within 0.05mm). Each step has a clear error control target and integrates multi-view features and edge information. The entire process follows an optimization logic from global to local and from low frequency to high frequency, ultimately outputting a high-precision depth map with an edge error of less than 0.05mm.

[0047] Step S5: Perform high-quality 3D reconstruction using explicit 3D Gaussian sputtering technology; S501: Input the fused feature map obtained in step S1, the camera pose parameters obtained in step S2, and the high-precision refined depth map obtained in step S4.

[0048] S502: Gaussian initialization; Based on the high-precision refined depth map and camera pose parameters, an initial Gaussian point cloud is generated in three-dimensional space.

[0049] The initial Gaussian point cloud contains 10 Gaussian points. 5 ~10 6 Each Gaussian point contains a mean, covariance, and weight parameters.

[0050] S503: Explicit 3D representation fusion; The fused feature map is mapped to a 3D space, and feature attributes are assigned to each Gaussian point to construct an explicit 3D feature field.

[0051] S504: Gaussian Sputtering and Optimization; The initial Gaussian point cloud is projected onto each view plane using a Gaussian kernel function. The photometric loss between the rendered image and the high-precision refined depth map obtained in step S4 is calculated. Combined with the sparsity loss of the initial Gaussian point cloud, the Gaussian point parameters are iteratively optimized. The number of iterations is set to 100-300, and the learning rate is set to 1e-4-1e-3.

[0052] S505: Outputs 3D reconstruction results, including 3D point cloud models and mesh models, with an accuracy of up to 0.1mm.

[0053] Step S6: Model training and inference process; (1) Training phase: Training dataset: A general multi-view dataset without pose annotation and fixed scene is used, covering various scenes such as indoor, outdoor and objects, with ≥100,000 images.

[0054] The loss function is as follows: Total loss ;in, Photometric loss is one of the core loss functions in the fields of computer vision and 3D reconstruction, and it is also the loss term with the highest weight (0.3) in the total loss function of this invention; This is the pose regression loss; As the core loss function related to depth, it has a weight of 0.2 in the total loss function. It is a key loss term that constrains the prediction accuracy of depth maps and ensures the consistency of depth information. It directly serves the entire process of initial depth generation and high-precision depth refinement, and is also an important guarantee for achieving millimeter-level 3D reconstruction. For rendering loss, This represents sparsity loss.

[0055] Training parameters: The optimizer is AdamW, the batch size is 8, the training epochs are 200, and the learning rate decay strategy is adopted (the learning rate is halved every 50 epochs).

[0056] In this invention, the training process does not rely on any 3D prior model, scene template, or ground truth pose or depth data, and is completely self-supervised training.

[0057] (2) Reasoning stage: Inputting unannotated multi-view images of any scene directly into the system requires no fine-tuning or additional parameter adjustments. The system's five core modules process the data linearly and output 3D reconstruction results. The reconstruction time for a single scene is 10-30 seconds, which can be adjusted according to the complexity of the scene.

[0058] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0059] 1. Configuration parameters; (1) Hardware environment: CPU is Intel Core i9-13900K, GPU is NVIDIA RTX 4090 (24GB video memory), and memory is 64GB DDR5.

[0060] (2) Software environment: The operating system is Ubuntu 22.04, the deep learning framework is PyTorch 2.0, the programming language is Python 3.9, and the CUDA version is 11.8.

[0061] (3) Input data: Select three sets of multi-view images without pose annotations for different scenes (Scene 1: Indoor desktop object, 5 views, resolution 1920×1080; Scene 2: Outdoor building, 7 views, resolution 1600×1200; Scene 3: Industrial parts, 4 views, resolution 2048×1536). All images have no pose annotations and no preset 3D prior information.

[0062] 2. Implementation steps; (1) Data preprocessing: Normalize the input multi-view image without pose annotation: normalize the pixel values ​​to [0,1], no other preprocessing operation is required.

[0063] (2) Model loading: Load the weight file of the trained, unconstrained, and prior generalizable 3D Gaussian sputtering model.

[0064] (3) Feature extraction: Input the preprocessed multi-view image into the multi-view feature extraction module and output a 256-dimensional fused feature map.

[0065] (4) Pose estimation: The fused feature map is input into the matching perception pose estimation network. After feature matching, pose regression and consistency verification, the camera pose parameters (rotation matrix and translation vector) of each view are output.

[0066] (5) Depth initialization and refinement: Based on the multi-view image and the fused feature map, the initial depth map is generated by the self-supervised depth initialization module, and the camera pose parameters are input into the depth refinement module to output a high-precision refined depth map.

[0067] (6) 3D reconstruction: The camera pose parameters, high-precision refined depth map and fused feature map are input into the explicit Gaussian sputtering reconstruction module. After 150 iterations of optimization, the 3D point cloud model and mesh model are output.

[0068] 3. Experimental Results and Data Analysis; (1) Reconstruction accuracy test: The absolute error (AE) and root mean square error (RMSE) were used as evaluation indicators. The results were compared with the closest existing technology as follows: ; (2) Generalization test: Select an unfamiliar scene (outdoor rock pile) that was not involved in the model training process, input 6 multi-view images without pose annotations, and the present invention can directly output the reconstruction results, AE=0.17mm, RMSE=0.21mm; when the prior art does not make fine adjustments in the same scene, AE=0.89mm, RMSE=1.05mm, the generalization performance of the present invention is more than 75% higher than that of the prior art.

[0069] (3) Stability test: Gaussian noise (variance = 0.05) was added to the input multi-view image. The pose estimation consistency error of the present invention was 0.042, while that of the prior art was 0.098. The stability of the present invention is improved by 57.1% compared with the prior art.

[0070] 4. This embodiment verifies through specific scenarios that the generalizable 3D Gaussian sputtering system and method of the present invention, which is free from pose constraints and prior knowledge, can achieve high-precision, high-stability, and high-generalization 3D reconstruction under the conditions of no pose annotation, no 3D prior knowledge, and no fine-tuning. It solves the core defects of the prior art and has practical application value.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge, characterized in that, Includes the following steps: Step S1: Input a multi-view image without pose annotation, extract multi-scale feature maps, perform weighted fusion of features at different scales, and obtain a fused feature map; Step S2: Estimate camera pose parameters using the fused feature map; Step S3: Initialize the depth using the pose-unannotated multi-view image and the fused feature map, and output the initial depth map; Step S4: Use a pyramid-shaped refinement structure and an edge-aware attention mechanism to obtain a high-precision refined depth map; Step S5: Perform high-quality 3D reconstruction using explicit 3D Gaussian sputtering technology.

2. The generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge as described in claim 1, characterized in that, In step S1, the improved ResNet-50 network is first used to perform layer-by-layer convolution and pooling operations on each input image to extract multi-scale feature maps. Then, the features at different scales are weighted and fused to obtain a fused feature map of the same dimension.

3. The generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge as described in claim 1, characterized in that, In step S2, a cross-correlation attention mechanism is first used to calculate the pixel-level matching similarity matrix between any two fused feature maps. Then, matching point pairs with a confidence score ≥ 0.8 are selected to construct a multi-view matching map. A convolutional neural network and a fully connected network are used to process the matching map to obtain the rotation matrix and translation vector of each matching map relative to the reference view. The pose transformation consistency error between any three fused feature maps is calculated. If the error is greater than the threshold, the confidence threshold is adjusted, and matching point pairs are re-selected until the consistency error is less than or equal to the threshold. Finally, the camera pose parameters that have passed the consistency verification are output.

4. The generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge as described in claim 1, characterized in that, In step S3, using pose-unlabeled multi-view images and fused feature maps, a photometric consistency loss function between views is constructed based on the principle of self-supervised multi-view stereo matching. The depth prediction network is optimized through unsupervised training, and the optimized depth prediction network outputs the initial depth map of each image.

5. The generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge as described in claim 4, characterized in that, The inter-view photometric consistency loss function is: , For the i-th multi-view image without pose annotation, For the j-th multi-view image without pose annotation, To obtain the pixel-level matching similarity matrix from feature map i to fused feature map j, This is the initial depth map.

6. The generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge as described in claim 1, characterized in that, In step S4, the initial depth map is decomposed into four scales using a pyramid-style refinement structure, and depth correction is performed on each scale, gradually optimizing from coarse to fine.

7. The generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge as described in claim 1, characterized in that, In step S4, edge information of the image is extracted from the fused feature map through the edge-aware attention mechanism, and the depth value of the edge region is given a higher optimization weight; the optimization objective is set to minimize the depth consistency loss.

8. The generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge as described in claim 1, characterized in that, The specific implementation process of step S5 is as follows: Based on the high-precision refined depth map and camera pose parameters, an initial Gaussian point cloud is generated in 3D space. The fused feature map is mapped to 3D space, and feature attributes are assigned to each Gaussian point to construct an explicit 3D feature field. The initial Gaussian point cloud is projected onto each view plane through a Gaussian kernel function. The photometric loss between the rendered image and the high-precision refined depth map obtained in step S4 is calculated. Combined with the sparsity loss of the initial Gaussian point cloud, the Gaussian point parameters are iteratively optimized. The 3D reconstruction results are output, including a 3D point cloud model and a mesh model.

9. The generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge as described in claim 1, characterized in that, It also includes step S6: training and inference, the specific implementation process of which is as follows: (1) Training phase: Training dataset: A general multi-view dataset without pose annotation and without fixed scene; Loss function: Total loss ; For luminance loss, For pose return loss, The core loss function is depth-dependent. For rendering loss, This is a sparsity loss; Training parameters: The optimizer is AdamW, the batch size is 8, the training epochs are 200, and the learning rate decay strategy is used; (2) Reasoning stage: The system can directly input multi-view images of any scene without pose annotations, without the need for fine-tuning or additional parameter adjustments. The system's linear pipeline processing outputs 3D reconstruction results.

10. A generalizable 3D Gaussian sputtering system without attitude constraints and prior knowledge, applied to the generalizable 3D Gaussian sputtering method without attitude constraints and prior knowledge as described in any one of claims 1-9, characterized in that, include: The multi-view feature extraction module is used to extract multi-scale feature maps, perform weighted fusion of features at different scales, and obtain a fused feature map with a unified dimension. A matching-aware pose estimation network is used to estimate camera pose parameters using fused feature maps; The self-supervised depth initialization module is used to initialize the depth using pose-unlabeled multi-view images and fused feature maps, and output an initial depth map. The depth refinement module is used to obtain high-precision refined depth maps by employing a pyramid-style refinement structure and an edge-aware attention mechanism. The explicit Gaussian sputtering reconstruction module is used to implement Gaussian initialization, explicit 3D representation fusion, and Gaussian sputtering and optimization to obtain 3D reconstruction results.

Images