Sparse sampling three-dimensional scene reconstruction method based on cognitive prior guidance

Abstract

The invention belongs to the technical field of artificial intelligence and three-dimensional scene reconstruction, and discloses a sparse sampling three-dimensional scene reconstruction method based on cognitive priori guidance. Comprising the following steps: data preprocessing and scene structure recovery; initializing a radiation field; extracting cognitive embedding and view priori generation; monocular depth constraint is carried out; reconstructing a three-dimensional scene; and calculating a loss function. According to the method, global cognition of a to-be-reconstructed scene is fully learned by constructing a scene feature cognition encoder, a pre-training generation model is guided to provide an effective training supplement view, geometric consistency of a radiation field is constrained by adopting depth information, distribution of Gaussian primitives is adjusted to solve the problem that a scene geometric structure is learned mistakenly, and the scene reconstruction efficiency is improved. And the quality of three-dimensional scene reconstruction in a sparse sampling scene is greatly improved. By using the method, scene global cognitive embedding can be effectively established, meanwhile, the geometric consistency of the scene is restrained, a compact and effective Gaussian radiation field is constructed, and the three-dimensional scene reconstruction quality under the sparse sampling condition is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and three-dimensional scene reconstruction technology, and particularly relates to a sparse sampling three-dimensional scene reconstruction method based on cognitive prior guidance. BACKGROUND

[0002] Three-dimensional scene reconstruction aims to recover the three-dimensional structure of the target scene from a set of two-dimensional images captured from the real world, which can be applied to the technical fields of unmanned driving, robot automatic navigation, medical image processing, and the key industries of digital economy such as virtual reality and augmented reality.

[0003] In recent years, three-dimensional Gaussian splatting (3D GS) has made breakthrough progress in the field of three-dimensional scene reconstruction. 3D GS uses three-dimensional Gaussian as a scene representation primitive, uses the learnable features of three-dimensional Gaussian to construct a Gaussian radiation field, and realizes real-time rendering of the scene through rasterization in the form of splatting, which has obvious advantages in rendering quality and training time. However, 3D GS is heavily dependent on the sampling density of the training samples on the scene, and it is often difficult to invest a large number of collection devices and deployment costs to meet the construction requirements of the Gaussian radiation field under real conditions, resulting in the problem of reconstruction distortion and partial omission of the scene when dealing with sparse data samples, which limits its practical application on mobile platforms and edge devices. Therefore, it is of significant application value to explore how to realize high-quality reconstruction of three-dimensional scenes under sparse sampling conditions.

[0004] Existing sparse-view 3D scene reconstruction methods can be roughly divided into two paradigms according to the Gaussian radiance field construction strategy: geometry structure optimization-based and generative enhancement-based. The geometry structure optimization-based method takes the geometric consistency of the scene as the optimization objective, adjusts the characteristics of the Gaussian cell in the radiance field by fully excavating valuable geometric constraints in sparse views. “Xu W, Gao H, Shen S, et al. Mvpgs: Excavating multi-view priors for gaussian splatting from sparse input views [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 203-220” predicts the depth consistency information of the scene through multi-view stereo vision technology, and then performs forward warping operation on the initialized radiance field to extract potential appearance information as the constraint of unseen views. “Li J, Zhang J, Bai X, et al. Dngaussian: Optimizing sparse-view 3D gaussian radiance fields with global-local depth normalization [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 20775-20785” proposes a global-local depth normalization method to learn the local depth variation in a scale-invariant manner, enhancing the ability of depth regularization to correct the geometric structure of the Gaussian radiance field. However, the geometry structure optimization-based method relies only on existing samples for feature matching and radiance field optimization, and is prone to local optimization under sparse sampling conditions. The incorrectly learned geometric structure will cause the Gaussian cell distribution to deviate from the real scene, resulting in scene structure rupture and limiting the scene reconstruction quality. To achieve good reconstruction of unseen views, the generative enhancement-based method is proposed. The generative enhancement-based method takes ensuring the visual quality of the scene as the optimization objective, synthesizes supplementary data for scene reconstruction through a pre-trained generative model, and guides the construction of the Gaussian radiance field."Zhong Y, Li Z, Chen D Z, et al. Taming Video Diffusion Prior with Scene-Grounding Guidance for 3D Gaussian Splatting from Sparse Inputs [C] / / Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 6133-6143" generates multi-view sequences using a video diffusion model, significantly increasing the training available instances of radiance field construction. "Li J, Wang F, Qu X, et al. EVPGS: Enhanced View Prior Guidance for Splatting-based Extrapolated View Synthesis [C] / / Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 16398-16407" proposes a coarse-to-fine two-stage radiance field construction framework that reduces view rendering artifacts due to insufficient view coverage by using prior knowledge of denoising diffusion models. Although the view prior is introduced for radiance field construction based on generative enhancement methods, the generative model has limitations in grasping the overall scene semantics, often failing to provide effective samples consistent with the scene semantics. Therefore, the current sparse sampling three-dimensional scene reconstruction method based on Gaussian radiance field will be negatively affected as follows: 1) insufficient view coverage leads to incorrect learning of scene geometry. 2) insufficient understanding of the overall scene semantics makes it difficult to effectively support the construction of the radiance field. Thus, the quality of scene reconstruction is affected.

[0005] To solve the above problems, the present application proposes a sparse sampling three-dimensional scene reconstruction method based on cognitive prior guidance, aiming to fully learn the global cognitive information of the scene to guide the generative model to provide effective view prior, and combine monocular depth information to constrain the geometric consistency of the radiance field, realizing high-quality three-dimensional scene reconstruction under the condition of sparse sampling. SUMMARY

[0006] In order to solve the above technical problems, the purpose of the present application is to provide a sparse sampling three-dimensional scene reconstruction method based on cognitive prior guidance. The method effectively extracts the global cognitive information of the scene by constructing a pose-feature joint cognitive encoder, provides reliable cognitive embedding for the generative model, and accurately generates view prior. At the same time, the monocular depth predictor is combined to constrain the geometric consistency of the scene and the distribution of Gaussian primitives, and a compact and effective Gaussian radiation field is constructed, which significantly improves the three-dimensional scene reconstruction quality under the condition of sparse sampling.

[0007] The technical scheme of the present application: a sparse sampling three-dimensional scene reconstruction method based on cognitive prior guidance, comprising the following steps:

[0008] Step 1: data preprocessing is performed on the training scene data set, and sparse samples are sampled for training;

[0009] Step 2: the pose information of the training scene data set is obtained by motion structure recovery, and a sparse three-dimensional point cloud is generated;

[0010] Step 3: the pose information and the sparse three-dimensional point cloud obtained in step 2 are used to construct a Gaussian radiation field, and the parameters and distribution positions of the three-dimensional Gaussian primitives are initialized;

[0011] Step 4: pseudo-view pose information is generated by uniformly sampling the view regions not covered by the training samples, and new views are rendered based on the Gaussian radiation field initialized in step 3;

[0012] Step 5: scene feature cognitive embedding is extracted through a scene feature cognitive encoder; the obtained scene feature cognitive embedding is used as an input prompt to guide the pre-trained diffusion model to generate view prior for the optimization of the Gaussian radiation field, and to assist the Gaussian primitives in the Gaussian radiation field to adjust the attribute parameters and distribution;

[0013] Step 6: the monocular depth predictor is used to predict the scene view depth information, and the consistency constraint is established with the depth views rendered under each view angle, so as to improve the geometric consistency of the radiation field;

[0014] Step 7: the construction of the Gaussian radiation field is optimized using a joint loss function, and the trained Gaussian radiation field is used for three-dimensional scene reconstruction under the condition of sparse sampling.

[0015] Further, the specific process of step 1 is: first, divide the training scene data set into a training set and a test set, and uniformly sample the image data in the training set as sparse samples for Gaussian radiation field construction.

[0016] Further, the specific process of step 2 is to calculate camera pose information from the pre-processed training set views in step 1 using Structure from Motion (SfM) and extract sparse 3D point clouds, with each sparse point being used as the initial point of a 3D Gaussian primitive.

[0017] Further, the specific process of constructing a Gaussian radiance field in step 3 is to use a 3D Gaussian as the basic unit of scene representation, which has anisotropic modeling capability, differentiable attribute parameters, and supports end-to-end training. The specific calculation formula is as follows:

[0018]

[0019] where G is a Gaussian function in 3D space, x represents the coordinates of a spatial point in 3D space, μ represents the center position of the Gaussian function, Σ represents the 3D covariance matrix, and ∑ is decomposed into a combination of rotation matrix R and scaling matrix S for easy adjustment and optimization of the Gaussian primitive:

[0020] ∑=RSS T R T (2)

[0021] Meanwhile, each Gaussian primitive also contains learnable parameters opacity α and spherical harmonic function coefficients c, where α ∈ [0, 1] and c is used to represent the directional color effect of the radiance field.

[0022] Further, the process of generating pseudo-view pose information in step 4 is as follows: first, select the two camera views with the closest Euclidean distance from the pose information of all training views obtained in step 2, take the average of the rotation quaternions of these two camera views to obtain the rotation direction of the new view, and then obtain the pseudo-view pose information and add it to the existing view pose set. The calculation formula is as follows:

[0023] P ′ =(t+ε,q),ε~(0,δ) (3)

[0024] q=avg(q i ,q j ) (4)

[0025] where P ′ represents the generated pseudo-view pose information, t represents the position of the camera, t ∈ P, P is the existing training view pose set, ε is the defined Gaussian noise disturbance, δ represents the standard deviation of the noise, avg represents the average calculation operation, q i ,q j represent the rotation quaternions of the i-th and j-th camera views, respectively, and q is the calculated rotation direction of the new view.

[0026] Further, the specific process of rendering a new view in step 4 is as follows: on the basis of the Gaussian radiation field constructed in step 3, for a pixel in the view to be reconstructed, the distances of all overlapping Gaussians along the observation direction to the pixel are calculated according to the depth information and a depth sorting list is formed, then the final color of the pixel is calculated by using alpha blending, and the calculation results of all pixels in the view are accumulated to obtain the rendered new view, and the calculation formula is as follows:

[0027]

[0028] ∑′=JWΣW T J T (7)

[0029] wherein represents the new view obtained after rendering, N represents a depth sorting list of all overlapping Gaussians along the observation direction, c i represents the spherical harmonic function coefficient of the i-th Gaussian element, a i represents the opacity of the i-th Gaussian element, a j represents the opacity of the remaining j-th Gaussian element, a i represents the result of the multiplication of the opacity of the i-th Gaussian element and the Gaussian element, a j represents the result of the multiplication of the opacity of the remaining j-th Gaussian element and the Gaussian element, x ′ and μ ′ i respectively represent the pixel coordinate projected to the two-dimensional space and the center position coordinate of the i-th Gaussian element, exp represents an exponential function, ∑ ′ represents the covariance matrix projected to the two-dimensional space, ∑ represents a three-dimensional covariance matrix, W is a view change matrix, and J is the Jacobian matrix of the projection transformation.

[0030] Further, the generation process of the scene feature recognition embedding in step 5 is as follows: first, the feature information of the new view rendered in step 4 is extracted by using convolution operation, and then the fused features are encoded by using a CLIP-based feature encoder to obtain CLIP image features, and the calculation formula is as follows:

[0031]

[0032] wherein, represents the new view rendered by step 3 under the pose p, Conv is a convolution function, and CLIP image is a frozen CLIP feature encoder, f clip represents the image features extracted by the CLIP feature encoder.

[0033] Subsequently, the cognitive bias correction is performed on the encoded CLIP image features through the cognitive adapter, and the final scene feature cognitive embedding is generated. In addition, the cognitive description of the training sample is generated by using BLIP2, which is used for the training supervision of the cognitive encoder, and the calculation formula is as follows:

[0034] E=MLP(CrossAttn(SelfAttn(Q),f Clip )) (9)

[0035] L=BLIP2 image (I) (10)

[0036] Wherein, MLP represents a multi-layer perceptron, CrossAttn represents a cross-attention mechanism, SelfAttn represents a self-attention mechanism, Q is a query vector, BLIP2 image represents a pre-trained BLIP2 model, I represents a sample in the training set applied to training, E is a scene feature cognitive embedding, and L is a generated training sample cognitive description.

[0037] Further, the view prior generation process in step 5 is that the generated scene feature cognitive embedding is input into a pre-trained diffusion model to generate a view prior for Gaussian radiation field optimization, which assists in adjusting the attribute parameters of the Gaussian primitive, and the calculation formula is as follows:

[0038]

[0039] Wherein I ref represents the generated supplementary view prior, StableDiffusion represents a pre-trained generative diffusion model, represents a new view rendered by step 3 under the pose p.

[0040] Further, the process of establishing consistency constraints in step 6 is as follows: first, the constructed Gaussian radiation field is rendered to obtain a depth view D est , and a monocular depth predictor is used to predict the depth of the new view rendered under each view angle to obtain a depth view D ras , and the calculation formula is as follows:

[0041]

[0042] Wherein D ras represents the depth view rendered under each view angle, D est represents the depth view predicted by the monocular depth predictor, Corr represents the Pearson correlation coefficient, Cov represents the calculation of the mean square error, Var represents the calculation of the standard deviation, and Mono represents the monocular depth prediction. This represents the new view rendered in step 3 at pose p.

[0043] Furthermore, the formula for calculating the joint loss of the Gaussian radiation field parameter optimization in step 7 is as follows:

[0044] Loss=λ1L1+λ2L D-SSIM +λ3L CE +λ4L regularization (14)

[0045]

[0046] L regularization =||Corr(D ras D est )||1 (18)

[0047] Where Loss is the overall loss, L1 is the pixel-level reconstruction loss, and L D-SSIM For structural similarity loss, L CE L is the semantic loss during cognitive encoder training. regularizaton Let ||.|| denote the geometric consistency loss, ||.|| represent the Euclidean norm, and I be the training view. This is the new view rendered in step 3. λ1, λ2, λ3, and λ4 are the weight coefficients of the corresponding loss functions.

[0048] The beneficial effects of this invention are as follows: This invention proposes a sparse sampling 3D scene reconstruction method based on cognitive prior guidance, which can fully understand the overall semantics of the 3D scene to construct a compact and effective Gaussian radiation field. Specifically, firstly, a scene feature cognitive encoder is proposed to establish a global cognitive embedding for views in various directions within the scene, fully mining the implicit prior knowledge in the pre-trained generative model to provide effective supplementary views for radiation field construction. Furthermore, a consistency constraint is established based on the depth alignment between the predicted depth of the monocular depth predictor and the rendered view, adjusting the distribution of Gaussian units. In addition, a pseudo-view generation method is employed to provide rich viewpoint information for radiation field construction to solve the problem of limited scene coverage, achieving high-quality 3D scene reconstruction under sparse sampling conditions. Attached Figure Description

[0049] Figure 1 This is a flowchart of the sparse sampling 3D scene reconstruction method based on cognitive prior guidance of the present invention;

[0050] Figure 2 This is an overall framework diagram of a specific embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the scene feature recognition encoder in this invention;

[0052] Figure 4 This is a flowchart of the test steps in a specific embodiment of the present invention. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The present invention includes, but is not limited to, the following embodiments.

[0054] like Figure 1 As shown, this invention provides a specific embodiment of a sparse sampling 3D scene reconstruction method based on cognitive prior guidance, the specific implementation process of which is as follows:

[0055] 1. Data preprocessing and scene structure restoration

[0056] The training scene dataset was compressed by downsampling at 4x and 8x respectively, and divided into training set and test set. The training set consists of 3 scene views, and the test set consists of 1 / 8 of the total number of training scene datasets. The SfM method was used to calculate the camera pose information from the training set views and extract sparse 3D point cloud, which was used as the basis for initializing the Gaussian radiation field.

[0057] 2. Radiation field initialization

[0058] like Figure 2 As shown, in the Gaussian radiation field initialization phase, initial Gaussian primitives are generated based on the 3D point cloud constructed from the training view and the corresponding pose information. Specifically, each point cloud in the 3D space is initialized as a 3D Gaussian function constituting a primitive of the radiation field, which has anisotropic modeling capabilities, differentiable attribute parameters, and supports end-to-end training. The calculation formula is shown below:

[0059]

[0060] Where G is the Gaussian function in three-dimensional space, x represents the coordinates of a point in three-dimensional space, μ represents the center position of the Gaussian function, and ∑ represents the three-dimensional covariance matrix. To facilitate the adjustment and optimization of Gaussian elements during the training phase, ∑ is decomposed into a combination of a rotation matrix R and a scaling matrix S, calculated as follows:

[0061] Σ=RSS T R T (20)

[0062] 3. Pseudo-view generation and view rendering

[0063] like Figure 2As shown, to ensure effective coverage of the scene to be reconstructed, a pseudo-viewpoint generation method is used to supplement pose information. First, from the pose information of all training views, the two camera views with the closest Euclidean distance are selected. The rotation quaternions of these two camera views are averaged to obtain the rotation direction of the new viewpoint, thus obtaining the pseudo-viewpoint pose information, which is then added to the existing view pose set. The calculation formula is as follows:

[0064] P′=(t+ε,q),ε~(0,δ) (21)

[0065] q = avg(q) i ,q j ) (twenty two)

[0066] Where P ′ The generated pseudo-viewpoint pose information is represented by t, where t represents the camera position, t∈P, P is the existing set of training view poses, ε is the defined Gaussian noise perturbation, δ represents the noise standard deviation, avg represents the operation of calculating the average value, and q i ,q j Let represent the rotation quaternions of the i-th and j-th camera viewpoints, respectively, and q be the calculated rotation direction of the new viewpoint.

[0067] Furthermore, to obtain the rendered views from various perspectives, for the constructed Gaussian radiation field, the distances from all overlapping Gaussians along the viewpoint to be reconstructed to each pixel in the view are calculated based on the depth information, and a depth-sorted list is formed. The final color of each pixel is calculated using alpha fusion, thus obtaining the rendered new view. The calculation formula is as follows:

[0068]

[0069] ∑′=JW∑W T J T (25)

[0070] in This represents the new view obtained after rendering, where N represents the depth-sorted list of all overlapping Gaussians along the viewing direction, and c... i Let α represent the coefficients of the spherical harmonic function of the i-th Gaussian element. i Let α represent the opacity of the i-th Gaussian element. j Let α represent the opacity of the remaining j-th Gaussian element. i ' represents the result of multiplying the opacity of the i-th Gaussian element by the Gaussian element, α j ' represents the result of multiplying the opacity of the remaining j-th Gaussian element by the Gaussian element, x ′ and μ ′ iLet Σ represent the pixel coordinates projected into two-dimensional space and the center coordinates of the i-th Gaussian element, respectively. Let exp represent the exponential function, and Σ represent the Σ-th Gaussian element. ′ Let Σ denote the covariance matrix projected onto two-dimensional space, Σ denote the three-dimensional covariance matrix, W be the view transformation matrix, and J be the Jacobian matrix of the projective transformation.

[0071] 4. Extracting cognitive embeddings and generating view priors

[0072] like Figure 2 and Figure 3 As shown, in the cognitive embedding extraction stage, a global cognitive embedding is established for views in various directions within the scene based on scene features by the cognitive encoder, in order to fully exploit the implicit prior knowledge in the pre-trained generative model. Specifically, firstly, a pseudo-viewpoint is generated based on all existing training view poses, and the pseudo-viewpoint is rendered. Then, scene cognition is performed using view pose information and image-level features. The encoder extracts the feature information of the rendered view, and a CLIP-based feature encoder is used to encode the image features. The feature encoder architecture uses CLIP ViT-Base-Patch32, and a cognitive adapter is used to correct cognitive biases in the encoded image features. The cognitive adapter consists of a 6-layer Transformer encoder structure, which is aligned to generate the final scene cognitive embedding. In addition, BLIP2 is used to generate cognitive descriptions of training samples for training supervision of the cognitive encoder. The calculation formula is as follows:

[0073]

[0074] E=MLP(CrossAttn(SelfAttn(Q),f Clip (27)

[0075] L = BLIP2 image (I) (28)

[0076] in This represents the new view rendered at pose p, where Conv is the convolution function with a kernel size of 3×3, and CLIP... image For the frozen CLIP feature encoder, f Clip This represents the image features extracted by the CLIP feature encoder. CrossAttn represents the cross-attention mechanism, SelfAttn represents the self-attention mechanism, Q is the query vector, the number of query vectors is set to 50, MLP represents a multilayer perceptron, and BLIP2 represents the multilayer perceptron. image Let represent the pre-trained BLIP2 model, I represent the samples in the training set used for training, E represent the learned scene feature cognitive embedding, and L represent the generated cognitive description of the training samples.

[0077] In the view prior generation stage, the generated scene cognition is embedded into the pre-trained diffusion model to generate supplementary view priors for radiation field optimization, which assists in adjusting the attribute parameters of Gaussian units. The calculation formula is as follows:

[0078]

[0079] Where I ref The generated supplementary view prior is represented by StableDiffusion, which represents the pre-trained generative diffusion model. The denoising steps are set to 20, and E represents the scene feature cognitive embedding. This represents the new view rendered at pose p.

[0080] 5. Monocular Depth Constraint

[0081] like Figure 2 As shown, the constructed Gaussian radiation field is first rendered using a depth view to obtain the depth view D. est Simultaneously, a monocular depth predictor is used to render new views from various perspectives. Depth prediction is performed to obtain the depth view D. ras The monocular depth predictor uses a depth estimation model based on the DPT-Hybrid architecture to establish the relative loss between the two depth views, calculated as follows:

[0082]

[0083] Where D ras D represents the depth view rendered from various perspectives. est This represents the depth view predicted by the monocular depth predictor. Corr represents the Pearson correlation coefficient, Cov represents the calculated root mean square error, Var represents the calculated standard deviation, and Mono represents the monocular depth prediction. This represents the new view rendered at pose p.

[0084] 6. Loss Function Calculation

[0085] Joint training is performed using image feature-level loss and geometric structure loss from 3D scene reconstruction. The attribute parameters of the Gaussian elements are adjusted, typically setting the total number of training iterations for constructing the Gaussian radiation field to 10,000. The calculation formula is as follows:

[0086] Loss=λ1L1+λ2L D-SSIM +λ3L CE +λ4L regularization (32)

[0087]

[0088] L regularization =||Corr(Dras D est )||1 (36)

[0089] Where Loss is the overall loss, L1 is the pixel-level reconstruction loss, and L D-SSIM For structural similarity loss, L CE L is the semantic loss during cognitive encoder training. regularizaton Let ||.|| denote the geometric consistency loss, ||.|| represent the Euclidean norm, and I be the training view. For the rendered view, E represents the scene feature cognitive embedding, L represents the cognitive description of the training samples, Corr represents the Pearson correlation coefficient, and D... ras For the rendered depth view, D est The depth view is predicted by the monocular depth predictor. λ1, λ2, λ3, and λ4 are the weight coefficients of the corresponding loss function, set to λ1 = 0.8, λ2 = 0.2, λ3 = 0.3, and λ4 = 0.1.

[0090] 7. 3D scene reconstruction for the test set

[0091] like Figure 4 As shown, for the test set view, its pose information is first extracted by SfM, and then it is input into the constructed Gaussian radiation field to render the view of the corresponding viewpoint to obtain the scene reconstruction result. Then, the construction quality of the Gaussian radiation field under sparse sampling conditions is judged by the effect of scene reconstruction. Generally, peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and learnable perceptual tile similarity (LPIPS) are used for quantitative evaluation.

[0092] In summary, this invention discloses a sparse sampling 3D scene reconstruction method based on cognitive prior guidance. This invention can fully learn the global cognition of the scene to be reconstructed to guide the pre-trained generative model to provide effective training supplementary views. Simultaneously, it effectively constrains the geometric consistency of the radiation field based on depth information, thereby adjusting the distribution of Gaussian units to solve the problem of incorrect learning of scene geometry, significantly improving the quality of 3D scene reconstruction in sparse sampling scenarios. First, this invention provides rich viewpoint information for radiation field construction through a pseudo-view generation strategy to avoid insufficient scene coverage. Then, it proposes a scene feature cognitive encoder to establish a global cognitive embedding of the scene, thereby accurately guiding the pre-trained generative model to create effective training supplementary views. Finally, it establishes consistency constraints based on the depth alignment between the predicted depth of the monocular depth predictor and the depth of the rendered view, optimizing and adjusting the distribution of Gaussian units to efficiently construct a compact and effective Gaussian radiation field, significantly improving the quality of 3D scene reconstruction in sparse sampling scenarios.

Claims

1. A sparse sampling 3D scene reconstruction method based on cognitive prior guidance, characterized in that, Includes the following steps: Step 1: Perform data preprocessing on the training scenario dataset to obtain sparse samples for training; Step 2: Motion structure recovery obtains the pose information of the training scene dataset and generates a sparse 3D point cloud; Step 3: Construct a Gaussian radiation field using the pose information obtained in Step 2 and the sparse 3D point cloud, and initialize the parameters and distribution positions of the 3D Gaussian primitives; Step 4: Uniformly sample and generate pseudo-viewpoint pose information in the viewpoint regions not covered by the training samples, and render and generate a new view based on the Gaussian radiation field obtained in Step 3. Step 5: Extract scene feature cognitive embeddings using a scene feature cognitive encoder; The obtained scene feature cognition is embedded as input prompts to guide the pre-trained diffusion model to generate view priors for Gaussian radiation field optimization, and to assist Gaussian elements in the Gaussian radiation field in adjusting attribute parameters and distribution. Step 6: Predict scene view depth information using a monocular depth predictor and establish consistency constraints with the depth views rendered from each viewpoint to improve the geometric consistency of the radiation field. Step 7: Optimize the construction of the Gaussian radiation field using the joint loss function. The trained Gaussian radiation field is used for 3D scene reconstruction in sparse sampling scenarios.

2. The sparse sampling 3D scene reconstruction method based on cognitive prior guidance according to claim 1, characterized in that, The specific process of step 1 is as follows: First, the training set and the test set are divided in the training scene dataset. Image data in the training set are uniformly sampled as sparse samples for the construction of the Gaussian radiation field.

3. The sparse sampling 3D scene reconstruction method based on cognitive prior guidance according to claim 1, characterized in that, The specific process of step 2 is as follows: using motion structure recovery to calculate camera pose information from the preprocessed training set view in step 1, extracting sparse 3D point cloud, and each sparse point is used as the initial point of 3D Gaussian primitives.

4. The sparse sampling 3D scene reconstruction method based on cognitive prior guidance according to claim 1, characterized in that, The specific process of constructing the Gaussian radiation field in step 3 is as follows: a three-dimensional Gaussian is used as the basic unit for scene representation. It has anisotropic modeling capabilities, its attribute parameters are differentiable, and it supports end-to-end training. The specific calculation formula is as follows: Where G is the Gaussian function in three-dimensional space, x represents the coordinates of a point in three-dimensional space, μ represents the center position of the Gaussian function, and Σ represents the three-dimensional covariance matrix. To facilitate the adjustment and optimization of Gaussian elements, Σ is decomposed into a combination of rotation matrix R and scaling matrix S: ∑=RSS T R T (2) In addition, each Gaussian element also contains a learnable parameter opacity α and a spherical harmonic function coefficient c, where α∈[0,1] and c is used to represent the directional color effect of the radiation field.

5. The sparse sampling 3D scene reconstruction method based on cognitive prior guidance according to claim 1, characterized in that, The process of generating pseudo-viewpoint pose information in step 4 is as follows: First, among all the pose information of the training views obtained in step 2, select the two camera views with the closest Euclidean distance, calculate the average of the rotation quaternions of these two camera views to obtain the rotation direction of the new viewpoint, and then obtain the pseudo-viewpoint pose information, which is then added to the existing viewpoint pose set. The calculation formula is as follows: P′=(t+ε,q),ε~(0,δ) (3) q=avg(q i ,q j ) (4) Where P′ represents the generated pseudo-viewpoint pose information, t represents the camera position, t∈P, P is the existing set of training view poses, ε is the defined Gaussian noise perturbation, δ represents the noise standard deviation, avg represents the operation of calculating the average value, and q i ,q j Let represent the rotation quaternions of the i-th and j-th camera viewpoints, respectively, and q be the calculated rotation direction of the new viewpoint.

6. The sparse sampling 3D scene reconstruction method based on cognitive prior guidance according to claim 1, characterized in that, The specific process of rendering and generating a new view in step 4 is as follows: Based on the Gaussian radiation field constructed in step 3, for each pixel in the view to be reconstructed, the distances to all overlapping Gaussians along the viewing direction are calculated according to the depth information, and a depth sorting list is formed. Then, the final color of the pixel is calculated using alpha fusion. The calculation results of all pixels in the view are accumulated to obtain the rendered new view. The calculation formula is as follows: ∑′=JWΣW T J T (7) in This represents the new view obtained after rendering, where N represents the depth-sorted list of all overlapping Gaussians along the viewing direction, and c... i Let α represent the coefficients of the spherical harmonic function of the i-th Gaussian element. i Let α represent the opacity of the i-th Gaussian element. j Let α represent the opacity of the remaining j-th Gaussian element. i ' represents the result of multiplying the opacity of the i-th Gaussian element by the Gaussian element, α j ' represents the result of multiplying the opacity of the remaining j-th Gaussian element by the Gaussian element, x' and μ' i Let represent the pixel coordinates projected onto the two-dimensional space and the center position coordinates of the i-th Gaussian element, respectively; exp represents the exponential function; ∑′ represents the covariance matrix projected onto the two-dimensional space; ∑ represents the three-dimensional covariance matrix; W is the view transformation matrix; and J is the Jacobian matrix of the projective transformation.

7. The sparse sampling 3D scene reconstruction method based on cognitive prior guidance according to claim 1, characterized in that, The specific process of generating the scene feature cognitive embedding in step 5 is as follows: First, convolutional operations are used to extract the feature information of the new view generated in step 4. Then, the fused features are encoded by a CLIP-based feature encoder to obtain CLIP image features. The calculation formula is as follows: in, This represents the new view rendered in step 3 at pose p, where Conv is the convolution function and CLIP is the new view. image For the frozen CLIP feature encoder, f Clip This represents the image features extracted by the CLIP feature encoder; Subsequently, the encoded CLIP image features are corrected for cognitive biases using a cognitive adapter to align and generate the final scene feature cognitive embedding. Furthermore, BLIP2 is used to generate cognitive descriptions of the training samples for training supervision of the cognitive encoder; the calculation formula is as follows: E=MLP(CrossAttn(SelfAttn(Q),f Clip )) (9) L=BLIP2 image (I) (10) Where MLP represents a multilayer perceptron, CrossAttn represents a cross-attention mechanism, SelfAttn represents a self-attention mechanism, Q is the query vector, and BLIP2... image Let I represent the pre-trained BLIP2 model, E represent the scene feature cognitive embedding, and L represent the generated cognitive description of the training samples.

8. The sparse sampling 3D scene reconstruction method based on cognitive prior guidance according to claim 1, characterized in that, The process of generating the view prior in step 5 is as follows: the generated scene feature cognition is embedded into the pre-trained diffusion model to generate the view prior for Gaussian radiation field optimization, which assists in adjusting the attribute parameters of the Gaussian elements. The calculation formula is as follows: Where I ref The first part represents the generated supplementary view prior, and the second part represents the pre-trained generative diffusion model. This represents the new view rendered in step 3 at pose p.

9. The sparse sampling 3D scene reconstruction method based on cognitive prior guidance according to claim 1, characterized in that, The process of establishing consistency constraints in step 6 is as follows: First, the constructed Gaussian radiation field is rendered with a depth view to obtain a depth view D. est Simultaneously, a monocular depth predictor is used to render new views from various perspectives. Depth prediction is performed to obtain the depth view D. ras The calculation formula is as follows: Where D ras D represents the depth view rendered from various perspectives. est This represents the depth view predicted by the monocular depth predictor. Corr represents the Pearson correlation coefficient, Cov represents the calculated root mean square error, Var represents the calculated standard deviation, and Mono represents the monocular depth prediction. This represents the new view rendered in step 3 at pose p.

10. The sparse sampling 3D scene reconstruction method based on cognitive prior guidance according to claim 1, characterized in that, The formula for calculating the joint loss in step 7, which optimizes the Gaussian radiation field parameters, is as follows: Loss=λ1L1+λ2L D-SSIM +λ3L CE +λ4L regularization (14) L regularization =||Corr(D ras ,D est )||1 (18) Where Loss is the overall loss, L1 is the pixel-level reconstruction loss, and L D-SSIM For structural similarity loss, L CE L is the semantic loss during cognitive encoder training. regularization Let ||.|| denote the geometric consistency loss, ||.|| represent the Euclidean norm, and I be the training view. This is the new view rendered in step 3; λ1, λ2, λ3, and λ4 are the weight coefficients of the corresponding loss functions.

Images