Sparse view synthesis method based on student's t-distribution and frequency-aware regularization

Abstract

This invention discloses a sparse view synthesis method based on the Student t-distribution and frequency-aware regularization. The invention employs collaborative modeling from a primitive representation layer, a low-frequency geometric constraint layer, and a high-frequency detail regularization layer: First, the Student t-distribution is introduced into the sparse view synthesis task for the first time, utilizing its adjustable heavy-tailed support properties to improve representation robustness under sparse supervision; second, wavelet decomposition is performed on the rendered depth map and monocular depth map, applying depth supervision only to the low-frequency subbands to stabilize the overall geometric structure and suppress high-frequency depth noise injection; third, a high-frequency mask is constructed based on the high-frequency components of the training image, and combined with the coverage area of ​​primitives or primitive clusters in the image plane, adaptively increasing the dropout probability of high-frequency regions, thereby suppressing local co-adaptation and new view artifacts. This invention can improve the quality, geometric consistency, and texture fidelity of new view synthesis under sparse view conditions while maintaining the advantages of explicit sputtering real-time rendering.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, 3D reconstruction and graphics rendering technology, and particularly relates to a sparse view synthesis method based on Student t-distribution and frequency-aware regularization. Background Technology

[0002] Novel view compositing aims to generate scene images with unobserved camera poses from finite input views, and is a key foundational technology in augmented reality, digital twins, robot perception, and 3D content production. In recent years, the 3D Gaussian sputtering method has achieved a good balance between reconstruction quality and real-time rendering speed through explicit 3D primitive representation and efficient sputtering rendering, and has become an important technical route for novel view compositing.

[0003] However, under sparse view conditions, the number of training images is limited, and existing 3D Gaussian sputtering methods tend to become overly dependent on a small number of training views, forming surface fitting rather than learning true representations of the scene, resulting in a significant decrease in the synthesis quality under unobserved viewpoints. Among existing methods, one type enhances overall geometric stability by introducing monocular depth supervision, artifact views, or geometric priors, but has limited ability to suppress artifacts and overfitting in high-frequency texture regions; another type alleviates local overfitting through regularization methods such as random dropout, but may destroy the already converged low-frequency geometric structure while suppressing artifacts.

[0004] Furthermore, frequency analysis reveals that degradation under a sparse perspective is not a single failure mode: low-frequency components are more likely to manifest as overall geometric underreconstruction and structural fragility, while high-frequency components are more likely to manifest as texture overfitting, artifacts, and flicker. Meanwhile, monocular depth priors are generally more reliable on low-frequency structures, but contain more high-frequency noise at edges and local details; directly applying depth supervision across the entire frequency band can easily inject erroneous depth information into the optimization process.

[0005] Furthermore, existing explicit representations based on Gaussian primitives, under sparse supervision, have a fixed effective support range and decay rapidly, easily causing the overall geometric representation to depend on fragile primitive overlap combinations. Therefore, there is an urgent need to propose a sparse view synthesis method that collaboratively models from a primitive representation layer, a low-frequency geometric constraint layer, and a high-frequency detail regularization layer, in order to simultaneously improve overall geometric stability and detail generalization ability. Summary of the Invention

[0006] The purpose of this invention is to address the problems of under-reconstruction of overall geometry, structural fragility, and overfitting of local high-frequency textures in existing 3D Gaussian sputtering methods under sparse input view conditions. This invention provides a sparse view synthesis method based on the Student t-distribution and frequency-aware regularization. This method introduces the Student t-distribution into the sparse view synthesis task and combines a depth supervision strategy based on wavelet low-frequency subbands and a frequency-aware importance discarding mechanism to achieve targeted intervention for degradation phenomena in different frequency bands.

[0007] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0008] In a first aspect, the present invention provides a sparse view synthesis method based on Student t-distribution and frequency-aware regularization, comprising the following steps:

[0009] S1. Obtain a sparse image set of the target scene, use an image-to-point-cloud generation algorithm to obtain sparse point clouds and corresponding camera pose parameters based on the sparse image set, and use a monocular depth estimation model to obtain a monocular depth map based on the sparse image set.

[0010] S2. Using sparse point cloud as geometric prior, initialize student t splash model containing multiple student t primitives. Use a low-frequency depth supervision strategy to iterate and update the student t splash model in multiple rounds. When the number of iterations reaches the preset iteration threshold, introduce a clustering high-frequency energy discard strategy to discard a portion of student t primitives that do not participate in the current round of splash rendering.

[0011] S3. In the inference phase, the camera pose of the new perspective is input into the trained student t splash model. Based on the learned student t primitive representation, the model generates a new view image under the new perspective through the differentiable student t rasterization rendering pipeline, thereby achieving high-quality synthesis of scene view under sparse perspective conditions.

[0012] Based on the above scheme, each step can be implemented in the following preferred manner.

[0013] As a preferred embodiment of the first aspect mentioned above, the specific process of S1 is as follows: taking multiple sparse images from the sparse image set as input, using a motion structure recovery algorithm to obtain sparse point clouds in the global coordinate system and camera pose parameters of each sparse image; inputting multiple sparse images from the sparse image set into a pre-trained monocular depth estimation model to obtain a monocular depth map of each sparse image.

[0014] As a preferred option in the first aspect mentioned above, the specific process of one iteration in the low-frequency deep supervision strategy of S2 is as follows:

[0015] S21. For each round of training, the sparse image selected is first generated by the student t-splash model under the camera pose parameters of S1, and the rendered depth map and rendered image are generated from the same viewpoint as the monocular depth map. Then, the low frequency components are extracted from the monocular depth map and rendered depth map of the sparse image by discrete wavelet transform, and the mean square error is calculated as the depth loss.

[0016] S22. Add the depth loss to the original loss of the student t-splash model to obtain the total loss. With minimizing the total loss function as the optimization objective, update the parameters of the student t-splash model through backpropagation to complete one round of iteration.

[0017] As a preferred embodiment of the first aspect mentioned above, in S21, low-frequency components of the two depth maps are extracted using first-level and second-level discrete wavelet transforms. The mean square error of the two low-frequency components is calculated under each level of discrete wavelet transform, and the calculated mean square errors are summed as the final depth loss. :

[0018] ;

[0019] in, This is the series index for the discrete wavelet transform; Calculation of mean square error; and These are the monocular depth map and the rendered depth map corresponding to the currently selected sparse image, respectively. for In the Low-frequency components obtained under discrete wavelet transform; for In the The low-frequency components obtained by discrete wavelet transform.

[0020] As a preferred embodiment of the first aspect mentioned above, in S2, the specific process of discarding a portion of student t-primitives through the clustering high-frequency energy drop-out strategy is as follows: First-order discrete wavelet transform is used to extract high-frequency components from the sparse image selected in this round of training, and the amplitude of the high-frequency component is taken as the high-frequency mask of the sparse image; then, the k-means algorithm is used to cluster based on the coordinates and opacity of the student t-primitives; subsequently, for each group obtained by clustering, the rendered image of the student t-primitives included in the group is multiplied element-wise by the high-frequency mask and summed, then divided by the sum of the elements of the rendered image to obtain the high-frequency energy of the group; the high-frequency energy of all groups is normalized by mean, which is used to calculate the dropout probability of each student t-primitive in each group.

[0021] As a preferred embodiment of the first aspect above, the calculation is performed according to the following formula. Dropout probability of each student's primitive t in each group :

[0022] ;

[0023] in, The preset base dropout probability; For the pre-set weights; For the first High-frequency energy of each group; , as well as These represent calculating the mean, finding the maximum value, and finding the minimum value, respectively. It is a set consisting of the high-frequency energies of all groups.

[0024] Secondly, the present invention provides a sparse view synthesis system based on Student t-distribution and frequency-aware regularization, comprising:

[0025] The data acquisition module is used to acquire a sparse image set of the target scene, use an image-to-point cloud generation algorithm to obtain sparse point clouds and corresponding camera pose parameters based on the sparse image set, and use a monocular depth estimation model to obtain a monocular depth map based on the sparse image set.

[0026] The model optimization module is used to initialize a student t splash model containing multiple student t primitives using sparse point cloud as geometric prior. A low-frequency depth supervision strategy is adopted to iterate and update the student t splash model in multiple rounds. When the number of iterations reaches the preset iteration threshold, a clustering high-frequency energy discard strategy is introduced to discard a part of the student t primitives and not participate in the splash rendering in this round.

[0027] The view generation module is used during the inference phase to input the camera pose of the new perspective into the trained student t splash model. Based on the learned student t primitive representation, the model generates a new view image from the new perspective through the differentiable student t rasterization rendering pipeline, thereby achieving high-quality synthesis of scene views under sparse view conditions.

[0028] Thirdly, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, enables the sparse view synthesis method based on student t-distribution and frequency-aware regularization as described in any of the solutions of the first aspect above.

[0029] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the sparse view synthesis method based on Student t-distribution and frequency-aware regularization as described in any of the solutions of the first aspect above.

[0030] Fifthly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0031] The memory is used to store computer programs;

[0032] The processor is configured to, when executing the computer program, implement the sparse view synthesis method based on student t-distribution and frequency-aware regularization as described in any of the first aspects above.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] First, this invention introduces the student t-distribution into the sparse view synthesis task, enhancing the robustness of explicit scene representation under sparse supervision and reducing the overall geometric structure's dependence on fragile primitive combinations. Second, this invention proposes a depth supervision strategy based on wavelet low-frequency subbands, improving overall geometric consistency and reducing the interference of high-frequency noise from depth priors on the optimization process. Third, this invention proposes a clustering high-frequency energy discarding strategy, specifically suppressing overfitting in high-frequency texture regions, reducing artifacts and flicker, while preserving as much of the converged low-frequency structure as possible. In summary, this invention can improve the quality of new view synthesis while maintaining the real-time rendering advantages of explicit sputtering methods. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0036] Figure 2 A schematic diagram illustrating the hollowing-out effect produced by scooping in the student t-distribution;

[0037] Figure 3 A schematic diagram of replacing Gaussian primitives and removing scooping operations for student t distribution under sparse view conditions;

[0038] Figure 4 A schematic diagram illustrating the visualization results of different methods on the LLFF dataset;

[0039] Figure 5 A schematic diagram illustrating the visualization results of different methods on the DTU dataset;

[0040] Figure 6 This is a system block diagram of the present invention;

[0041] Figure 7 This is a schematic diagram of a computer electronic device provided by the present invention. Detailed Implementation

[0042] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0043] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0044] This invention proposes a sparse view synthesis method based on Student's t-distribution and frequency-aware regularization. This method aims to construct a frequency-domain-guided sparse reconstruction framework, which can improve upon existing 3D Gaussian splashing reconstruction techniques. The core idea is that while existing spatial domain methods have achieved good results, they mostly operate globally, neglecting frequency domain characteristics. Furthermore, it is noted that the original Gaussian splashing method is highly sensitive to perturbations in the low-frequency region, which may be related to the inherent limitations of 3D Gaussian distributions. Therefore, this invention replaces the 3D Gaussian distribution with a Student's t-distribution.

[0045] The sparse view synthesis method described above generally includes: point cloud and camera pose acquisition, monocular depth map acquisition, student t-cell initialization, low-frequency depth supervision, and clustering high-frequency energy discarding. Point cloud and camera pose acquisition, monocular depth map acquisition, and student t-cell initialization belong to the initialization phase, aiming to construct a high-quality geometric prior; low-frequency depth supervision and clustering high-frequency energy discarding belong to the optimization and regularization phase, aiming to suppress overfitting and improve reconstruction accuracy.

[0046] like Figure 1 As shown, in a preferred embodiment of the present invention, the sparse view synthesis method based on Student t-distribution and frequency-aware regularization includes the following steps S1 to S3. The specific implementation process of each step is described in detail below.

[0047] S1. Obtain a sparse image set of the target scene, use an image-to-point-cloud generation algorithm to obtain sparse point clouds and corresponding camera pose parameters based on the sparse image set, and use a monocular depth estimation model to obtain a monocular depth map based on the sparse image set.

[0048] It should be noted that the specific process of S1 in this invention is as follows: multiple sparse images in the sparse image set are used as input, and the Structure from Motion (SfM) algorithm is used to obtain the sparse point cloud in the global coordinate system and the camera pose parameters of each sparse image; multiple sparse images in the sparse image set are input into a pre-trained monocular depth estimation model to obtain the monocular depth map of each sparse image.

[0049] In step S1 of this embodiment, the image-to-point cloud generation algorithm can employ existing VGGT Durst3R models or SfM algorithms, etc. This embodiment specifically uses the SfM algorithm for point cloud generation. The monocular depth estimation model can employ existing DPT, VGGT, etc., or other models used to estimate image pixel depth can be selected according to actual needs, without limitation in this invention.

[0050] Furthermore, the pre-acquired sparse image set contains a total of Sparse image, For the number of sparse images, the th Zhang's sparse image is represented as Each sparse image is input into COLMAP and aggregated to obtain a sparse point cloud in the global coordinate system. and the camera pose parameters of each sparse image, the first The camera pose parameters of the sparse image are represented as follows: , No. The monocular depth map of the sparse image is .

[0051] S2. Using sparse point cloud as geometric prior, initialize a student t-splatting and scooping (SSS) model containing multiple student t-primitives. Employ a low-frequency depth supervision strategy to iterate and update the student t-splatting model in multiple rounds. When the number of iterations reaches a preset iteration threshold, introduce a clustering high-frequency energy discarding strategy to discard a portion of student t-primitives that do not participate in the current round of splatting rendering.

[0052] It should be noted that in this invention, the aforementioned student t primitive includes parameters such as coordinates, covariance matrix, opacity, and spherical harmonic coefficients. The contribution of each student t primitive in the student t-splash model is calculated from the Student's t-distribution. The implementation of the student t-splash model here is existing technology. It replaces the traditional Gaussian distribution with the statistically derived Student's t-distribution and innovatively introduces the concept of "negative density" (Scooping), thereby significantly improving rendering quality, detail representation, and parameter efficiency while maintaining real-time rendering speed. During the initialization of the student t-splash model, the initial coordinates of each student t primitive are obtained from the coordinates of the sparse point cloud, while other parameters (such as covariance matrix, opacity, and spherical harmonic coefficients) are obtained through random initialization.

[0053] It should be noted that the specific process of one iteration in the low-frequency depth supervision strategy of this invention S2 is as follows:

[0054] S21. For each round of training, the sparse image selected is first generated by the student t-splash model under the camera pose parameters in S1, using a rendered depth map and a rendered image from the same viewpoint as the monocular depth map. Then, discrete wavelet transform is used to extract the monocular depth map from the sparse image. and rendering depth map Low-frequency components are extracted, and the mean square error is calculated as the depth loss.

[0055] It should be noted that in S21 of this invention, low-frequency components of two depth maps are extracted using first-level and second-level discrete wavelet transforms. The mean square error of the two low-frequency components is calculated under each level of discrete wavelet transform, and the sum of the calculated mean square errors is used as the final depth loss. :

[0056]

[0057] in, This is the series index for the discrete wavelet transform; Calculation of mean square error; and These are the monocular depth map and the rendered depth map corresponding to the currently selected sparse image, respectively. for In the Low-frequency components obtained under discrete wavelet transform; for In the The low-frequency components obtained by discrete wavelet transform.

[0058] S22. Add the depth loss to the original loss of the student t-splash model to obtain the total loss. With minimizing the total loss function as the optimization objective, update the parameters of the student t-splash model through backpropagation to complete one round of iteration.

[0059] In this embodiment, the above total loss It is expressed as follows:

[0060]

[0061] in, Used to control the weights of depth loss; The original loss inherent in the student t-splash model.

[0062] It should be noted that, in S2 of this invention, the specific process of discarding a portion of student t primitives through the clustering high-frequency energy discarding strategy is as follows: using first-order discrete wavelet transform to select sparse images from the current training round. High-frequency components are extracted, and the amplitude of these high-frequency components is used as the high-frequency mask of the sparse image. Then, the k-means algorithm is used to cluster the student t-cells based on their coordinates and opacity. Subsequently, for each cluster, the rendered image of the student t-cells included in the cluster is multiplied element-wise by the high-frequency mask and summed. This sum is then divided by the sum of the elements of the rendered image to obtain the high-frequency energy of the cluster. The high-frequency energy of all clusters is normalized by mean and used to calculate the dropout probability of each student t-cell in each cluster.

[0063] In this embodiment, when performing clustering using the k-means algorithm, the coordinates of student t primitives need to be normalized using min-max to make their range similar to their opacity. The sum of squared distances between the s-th student t primitive and its cluster center is... It consists of the sum of two terms: the first term is the squared error between the coordinates of the s-th student t primitive and the coordinates of the cluster center; the second term is the squared error between the opacity of the s-th student t primitive and the opacity of the cluster center, specifically expressed as:

[0064]

[0065] in, , , The three components of the coordinates of the t-th student are given; , , These are the three components of the cluster center coordinates; Let t be the opacity of the s-th student's primitive t; Opacity of cluster centers. Corresponding to the first item, it is calculated from the three components of the coordinates of the t primitive of the s-th student and the three components of the coordinates of the cluster center; This corresponds to the second item.

[0066] Based on the k-means clustering described above, this embodiment calculates the high-frequency energy for each group. Taking the first group as an example... Taking a group as an example, its high-frequency energy Represented as:

[0067]

[0068] in, For element-wise multiplication; for The amplitude, i.e., the sparse image selected in this round of training. High-frequency mask, ; Sparse images selected for this training round High-frequency components; For the first The rendered image of the student t primitives included in each group.

[0069] Finally, the present invention calculates the first high-frequency energy using the above-mentioned method. Dropout probability of each student's primitive t in each group :

[0070]

[0071] in, The preset base dropout probability; For the pre-set weights; , as well as These represent calculating the mean, finding the maximum value, and finding the minimum value, respectively. It is a set consisting of the high-frequency energies of all groups.

[0072] It should be noted that, for all student t primitives within the same group, this invention assigns the same dropout probability to these student t primitives. The dropout probability of each student t primitive is... This refers to the probability that the student's t primitive will not participate in the splash rendering during the rendering process. .

[0073] S3. In the inference phase, the camera pose of the new perspective is input into the trained student t splash model. Based on the learned student t primitive representation, the model generates a new view image under the new perspective through the differentiable student t rasterization rendering pipeline, thereby achieving high-quality synthesis of scene view under sparse perspective conditions.

[0074] It should also be noted that, unlike the original student t-splatter model, this invention abandons scooping. The reason is that, under sparse view conditions, due to insufficient supervision, scooping is prone to creating hollowing-out effects in incorrect locations, resulting in holes in the new view, such as... Figure 2 As shown. Therefore, this invention uses the student t distribution under sparse perspective conditions to replace the Gaussian primitive and removes the scooping operation, with the following effect: Figure 3 As shown.

[0075] To better demonstrate the specific implementation and technical effects of the present invention, the sparse view synthesis method based on student t-distribution and frequency-aware regularization shown in steps S1 to S3 of the above preferred implementation is applied to a specific example.

[0076] Example

[0077] The specific implementation process of the sparse view synthesis method based on student t-distribution and frequency-aware regularization used in this embodiment is as described above and will not be repeated here.

[0078] This invention underwent extensive experimental validation on the publicly available datasets LLFF and DTU, focusing on evaluating its performance improvement compared to existing reconstruction methods (such as DNGaussian, FSGS, DropGaussians, etc.) and the original Student's t-splash method (SSS). Experiments were conducted using three training views at full resolution, with testing performed on other views. Evaluation metrics included Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Perceptual Loss (LPIPS), and Average Error (AVGE).

[0079] The AVGE calculation method is as follows:

[0080]

[0081] As shown in Tables 1 and 2, compared with existing methods, the present invention achieves significant improvements in all reconstruction metrics under sparse views, far surpassing the original methods. This represents the difference between the index of this invention and the suboptimal value. The qualitative comparison results are as follows: Figure 4 , Figure 5As shown in the figure, not only is the reconstruction effect of the method of this invention demonstrated, but it also compares the reconstruction results of various existing advanced 3D Gaussian reconstruction methods (3DGS, DNGaussian, FSGS, DropGaussians, CoR-GS, DropoutGS, SSS) under the same sparse view. It can be seen that the aforementioned existing methods all exhibit varying degrees of failure under sparse views, such as artifacts and loss of high-frequency details. In contrast, the reconstructed view from the novel perspective of the method of this invention shows significant advantages in geometric integrity, texture restoration, and artifact suppression, effectively addressing the shortcomings of the original methods in sparse scenes. These results fully demonstrate that the present invention possesses superior sparse view reconstruction capabilities.

[0082] Table 1. Test Data Transformation Results of the LLFF Dataset

[0083]

[0084] Table 2. Results of DTU Dataset Test Data Generation

[0085]

[0086] It should also be noted that the sparse view synthesis method based on Student's t-distribution and frequency-aware regularization in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a sparse view synthesis system based on Student's t-distribution and frequency-aware regularization, corresponding to the sparse view synthesis method based on Student's t-distribution and frequency-aware regularization provided in the above embodiments, such as... Figure 6 As shown, it includes:

[0087] The data acquisition module is used to acquire a sparse image set of the target scene, use an image-to-point cloud generation algorithm to obtain sparse point clouds and corresponding camera pose parameters based on the sparse image set, and use a monocular depth estimation model to obtain a monocular depth map based on the sparse image set.

[0088] The model optimization module is used to initialize a student t splash model containing multiple student t primitives using sparse point cloud as geometric prior. A low-frequency depth supervision strategy is adopted to iterate and update the student t splash model in multiple rounds. When the number of iterations reaches the preset iteration threshold, a clustering high-frequency energy discard strategy is introduced to discard a part of the student t primitives and not participate in the splash rendering in this round.

[0089] The view generation module is used during the inference phase to input the camera pose of the new perspective into the trained student t splash model. Based on the learned student t primitive representation, the model generates a new view image from the new perspective through the differentiable student t rasterization rendering pipeline, thereby achieving high-quality synthesis of scene views under sparse view conditions.

[0090] It is understood that the sparse view synthesis method based on Student's t-distribution and frequency-aware regularization described in S1-S3 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the sparse view synthesis method based on Student's t-distribution and frequency-aware regularization provided in the above embodiments. This product includes a computer program / instruction that, when executed by a processor, can implement the sparse view synthesis method based on Student's t-distribution and frequency-aware regularization as described in the above embodiments.

[0091] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the sparse view synthesis method based on Student t-distribution and frequency-aware regularization provided in the above embodiments, such as... Figure 7 As shown, it includes a memory and a processor;

[0092] The memory is used to store computer programs;

[0093] The processor is configured to implement the sparse view synthesis method based on student t-distribution and frequency-aware regularization in the above embodiments when executing the computer program.

[0094] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0095] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the sparse view synthesis method based on Student t-distribution and frequency-aware regularization provided in the above embodiments. The storage medium stores a computer program that, when executed by a processor, can implement the sparse view synthesis method based on Student t-distribution and frequency-aware regularization in the above embodiments.

[0096] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by a processor, which can perform the aforementioned steps S1 to S3.

[0097] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0098] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0099] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0100] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A sparse view synthesis method based on Student t-distribution and frequency-aware regularization, characterized in that, Includes the following steps: S1. Obtain a sparse image set of the target scene, use an image-to-point-cloud generation algorithm to obtain sparse point clouds and corresponding camera pose parameters based on the sparse image set, and use a monocular depth estimation model to obtain a monocular depth map based on the sparse image set. S2. Using sparse point cloud as geometric prior, initialize student t splash model containing multiple student t primitives. Use a low-frequency depth supervision strategy to iterate and update the student t splash model in multiple rounds. When the number of iterations reaches the preset iteration threshold, introduce a clustering high-frequency energy discard strategy to discard a portion of student t primitives that do not participate in the current round of splash rendering. S3. In the inference phase, the camera pose of the new perspective is input into the trained student t splash model. Based on the learned student t primitive representation, the model generates a new view image under the new perspective through the differentiable student t rasterization rendering pipeline, thereby achieving high-quality synthesis of scene view under sparse perspective conditions.

2. The sparse view synthesis method based on student t-distribution and frequency-aware regularization as described in claim 1, characterized in that, The specific process of S1 is as follows: multiple sparse images in the sparse image set are used as input, and the motion structure recovery algorithm is used to obtain the sparse point cloud in the global coordinate system and the camera pose parameters of each sparse image; multiple sparse images in the sparse image set are input into the pre-trained monocular depth estimation model to obtain the monocular depth map of each sparse image.

3. The sparse view synthesis method based on Student t-distribution and frequency-aware regularization as described in claim 1, characterized in that, In S2's low-frequency deep supervision strategy, the specific process of one iteration is as follows: S21. For each round of training, the sparse image selected is first generated by the student t-splash model under the camera pose parameters of S1, and the rendered depth map and rendered image are generated from the same viewpoint as the monocular depth map. Then, the low frequency components are extracted from the monocular depth map and rendered depth map of the sparse image by discrete wavelet transform, and the mean square error is calculated as the depth loss. S22. Add the depth loss to the original loss of the student t-splash model to obtain the total loss. With minimizing the total loss function as the optimization objective, update the parameters of the student t-splash model through backpropagation to complete one round of iteration.

4. The sparse view synthesis method based on Student t-distribution and frequency-aware regularization as described in claim 3, characterized in that, In S21, first-level and second-level discrete wavelet transforms are used to extract low-frequency components from two depth maps. The mean square error of the two low-frequency components is calculated under each level of discrete wavelet transform, and the sum of these mean square errors is used as the final depth loss. : ； in, This is the series index for the discrete wavelet transform; Calculation of mean square error; and These are the monocular depth map and the rendered depth map corresponding to the currently selected sparse image, respectively. for In the Low-frequency components obtained under discrete wavelet transform; for In the The low-frequency components obtained by discrete wavelet transform.

5. The sparse view synthesis method based on Student t-distribution and frequency-aware regularization as described in claim 3, characterized in that, In S2, the specific process of discarding a portion of student t-primitives using the clustering high-frequency energy drop-out strategy is as follows: First-order discrete wavelet transform is used to extract high-frequency components from the sparse image selected in this training round, and the amplitude of these high-frequency components is taken as the high-frequency mask of the sparse image; then, the k-means algorithm is used to cluster the student t-primitives based on their coordinates and opacity; subsequently, for each cluster obtained, the rendered image of the student t-primitives included in the cluster is multiplied element-wise by the high-frequency mask and summed, then divided by the sum of all elements of the rendered image to obtain the high-frequency energy of that cluster; the high-frequency energy of all clusters is normalized by mean, which is used to calculate the dropout probability of each student t-primitive in each cluster.

6. The sparse view synthesis method based on Student t-distribution and frequency-aware regularization as described in claim 5, characterized in that, Calculate the first one according to the following formula Dropout probability of each student's primitive t in each group : ； in, The preset base dropout probability; For the pre-set weights; For the first High-frequency energy of each group; , as well as These represent calculating the mean, finding the maximum value, and finding the minimum value, respectively. It is a set consisting of the high-frequency energies of all groups.

7. A sparse view synthesis system based on Student t-distribution and frequency-aware regularization, characterized in that, include: The data acquisition module is used to acquire a sparse image set of the target scene, use an image-to-point cloud generation algorithm to obtain sparse point clouds and corresponding camera pose parameters based on the sparse image set, and use a monocular depth estimation model to obtain a monocular depth map based on the sparse image set. The model optimization module is used to initialize a student t splash model containing multiple student t primitives using sparse point cloud as geometric prior. A low-frequency depth supervision strategy is adopted to iterate and update the student t splash model in multiple rounds. When the number of iterations reaches the preset iteration threshold, a clustering high-frequency energy discard strategy is introduced to discard a part of the student t primitives and not participate in the splash rendering in this round. The view generation module is used during the inference phase to input the camera pose of the new perspective into the trained student t splash model. Based on the learned student t primitive representation, the model generates a new view image from the new perspective through the differentiable student t rasterization rendering pipeline, thereby achieving high-quality synthesis of scene views under sparse view conditions.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the sparse view synthesis method based on student t-distribution and frequency-aware regularization as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the sparse view synthesis method based on student t-distribution and frequency-aware regularization as described in any one of claims 1 to 6.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the sparse view synthesis method based on student t-distribution and frequency-aware regularization as described in any one of claims 1 to 6.

Images