3D reconstruction method of Gaussian radiation field based on geometric distillation of pre-trained optical flow model

Abstract

The present invention provides a Gaussian radiation field 3D reconstruction method based on geometric distillation of a pre-trained optical flow model, comprising the following steps: step 1, inputting a training picture set and a corresponding pose matrix, an intrinsic parameter matrix of a camera and a pre-trained matching model; initializing the Gaussian radiation field at the same time; step 2, executing the optimization process of the Gaussian radiation field; step 3, rendering a picture of a viewing angle; step 4, calculating the corresponding position of a pixel of a training viewing angle to a neighboring viewing angle; step 5, obtaining a priori optical flow; step 6, optimizing a 3D Gaussian radiation field; step 7, after the loss decreases, the optimization ends and the final Gaussian radiation field is obtained. The present invention significantly improves the geometric reconstruction accuracy of the Gaussian radiation field by utilizing the optical flow model matching information of an unobserved viewing angle during the training process. Compared with other geometric prior information, the matching flow prior information combined with the pose input can restore the absolute scale of the scene, thereby providing more reliable guidance for geometric optimization.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and in particular relates to a Gaussian radiation field three-dimensional reconstruction method based on geometric distillation of a pre-trained optical flow model. Background Art

[0002] 3D Gaussian Splatting (3DGS) was first proposed in 2023. This method introduces three-dimensional Gaussian distributed particles to represent the radiation field information in the scene, thereby achieving efficient real-time rendering and 3D reconstruction. Since its proposal, 3DGS has been rapidly and widely used, especially in various 3D reconstruction and rendering tasks. Its application areas cover the reconstruction and rendering of static to dynamic scenes. 3DGS is used to synthesize new perspectives in dynamic scenes, showing its powerful performance in complex dynamic scenes.

[0003] In addition to new perspective synthesis, 3DGS has also made significant progress in other areas. In terms of mesh reconstruction, by combining 3DGS technology with surface-aligned Gaussian particles, more efficient 3D mesh reconstruction and high-quality mesh rendering can be achieved.

[0004] However, despite its excellent performance in many fields, 3DGS still faces challenges in certain specific situations. Especially in areas with sparse viewpoints (such as complex indoor environments or boundless outdoor scenes), due to the limited number of input viewpoints, radiation field optimization is prone to overfitting, resulting in reduced accuracy of geometric reconstruction. This overfitting problem usually leads to unreliable geometric representation and affects the quality of reconstruction. Therefore, how to further improve the robustness and accuracy of 3DGS under sparse viewpoints remains an important direction for future research.

[0005] In order to alleviate the overfitting problem, recent research work has focused on integrating geometric prior information into the optimization process of the radiation field represented by 3D Gaussian points. These studies attempt to improve the generalization ability of the model by introducing additional geometric constraints to avoid overfitting in the case of sparse viewpoints. For example, regularization of sensor depth information and surface normals can be introduced in the reconstruction process to help guide the optimization of the radiation field and ensure the accuracy of the geometric structure. This method reduces the geometric uncertainty caused by sparse viewpoints by using the depth data collected by the sensor and the calculated normal information as constraints. However, directly obtaining high-precision sensor depth data is often costly, especially in large-scale or dynamic scenes, which is difficult to achieve. At the same time, there are also problems with depth maps generated by relying on pre-trained deep models, mainly manifested as scale ambiguity, that is, the depth maps generated by the model often cannot provide accurate absolute depth information, thereby limiting its effectiveness in optimizing scene geometry.

[0006] Similarly, although pre-trained surface normal models can provide additional information about scene geometry, their output normals are usually limited to recovering local geometry and lack understanding of the overall scale of the scene. This limitation makes normal constraints difficult to apply in all scenarios, especially when absolute scale information is required. Therefore, although these geometric prior information can alleviate the overfitting problem to a certain extent, how to effectively obtain and use these priors, especially without relying on expensive sensors, remains one of the main challenges facing current technology.

[0007] Compared with monocular prior information, matching flow prior information combined with camera pose input can effectively restore the absolute scale of scene geometry. This is because the matching flow prior can infer the depth and geometric structure of the scene through the relative motion between viewpoints, thus avoiding the scale ambiguity problem common in monocular methods. Based on this feature, an intuitive improvement method is to integrate the matching prior information between different training viewpoints into the training process of the radiation field, and improve the reconstruction accuracy through the geometric relationship between viewpoints.

[0008] Specifically, the matching flow prior information can provide geometric consistency between perspectives, so that the observation information from different perspectives complement each other, thereby better optimizing the geometric representation of the radiation field. This method is particularly suitable for multi-perspective scene reconstruction tasks and can effectively improve the ability to restore geometric details. However, when there are large differences between the training perspectives (for example, the perspectives are far apart or the observed scene areas do not overlap), this matching method may cause significant matching errors. Due to the large difference in perspectives, it may be difficult for the algorithm to accurately estimate the geometric relationship between perspectives, resulting in the failure of the matching flow, which affects the reconstruction effect. This matching error not only causes geometric errors, but may also cause incorrect transmission of color or lighting information in the radiation field, further reducing the overall rendering quality.

[0009] In addition, how to effectively integrate matching prior information from algorithmically controllable unobserved viewpoints into the radiation field geometry optimization process remains an underexplored research area. Matching prior information from unobserved viewpoints is usually obtained by inferring and extrapolating the existing observation data of the scene, which can provide additional constraints for the optimization of radiation field geometry. However, the geometric information of these viewpoints is often not precise enough or has large uncertainties, so directly applying it to the geometry optimization process may lead to new errors. In addition, how to balance the inferred information from unobserved viewpoints with the precise information from observed viewpoints is also an issue worthy of further exploration.

[0010] In summary, although matching flow prior information provides a powerful means to improve the accuracy of geometric reconstruction, its application in the face of large differences in viewpoints or when no viewpoint inference information is observed is still challenging. Further research is needed to solve these challenges in order to achieve more robust and high-precision geometric reconstruction in multi-view and sparse view situations. Summary of the invention

[0011] Purpose of the invention: The technical problem to be solved by the present invention is to provide a Gaussian radiation field 3D reconstruction method based on geometric distillation of a pre-trained optical flow model in view of the deficiencies of the prior art, comprising the following steps:

[0012] Step 1: Input training image set And the corresponding pose matrix , the camera's intrinsic parameter matrix K and a pre-trained matching model ; Initialize the Gaussian radiation field at the same time ,in is the i-th Gaussian point, represents the i-th training picture, B represents the total number of training pictures, Represents the pose matrix of the i-th training picture, and the pose matrix is ​​the viewing angle; are the parameters of the pre-trained matching model; the pre-trained matching model is an optical flow model; , , , , , They correspond to the rotation, scale, color, position and opacity of the i-th Gaussian point respectively;

[0013] Step 2: Execute the optimization process of Gaussian radiation field: First, randomly select a perspective from the perspective of the training image. As the current training perspective, from this perspective Sampling an unobserved view in the neighborhood of ; The range of the neighborhood is a circle with a radius of r;

[0014] Step 3: Rendering perspective Pictures , render the current training perspective Pictures and the corresponding geometric depth map;

[0015] Step 4: Through the current training perspective Depth , and the current training perspective With perspective The pose transformation , calculate the corresponding position of the training view pixel to the neighborhood view, and get the geometric optical flow ;

[0016] Step 5: Change the viewing angle Pictures And the training picture input pre-trained matching model , get the prior optical flow ;

[0017] Step 6: Use prior optical flow To supervise the geometric optical flow , and get the optical flow loss , combine color loss and structural similarity loss, perform gradient back propagation, and optimize the 3D Gaussian radiation field; return to step 2 until the optimization converges, and the overall optimization loss function L is:

[0018] ;

[0019] in, is the structural similarity loss weight, is the optical flow loss weight, represents the color loss function, represents the structural similarity loss function, is the surface normal consistency loss, is the surface normal consistency loss weight;

[0020] Step 7: After the loss decreases, the optimization ends and the final Gaussian radiation field is obtained.

[0021] In step 2, ; For perspective The average depth of the image, f is the focal length, A user-defined parameter.

[0022] Step 3 includes: The expression of Gaussian radiation sphere is:

[0023] ,

[0024] in is the distribution of the i-th Gaussian radiation sphere at the spatial position X, is the center position of the i-th Gaussian radiation sphere, e is a natural constant, T represents the matrix transpose, By rotation and scale Decide: ; where S is the scale The parameters obtained by diagonalization, R is the quaternion rotation The resulting rotation matrix.

[0025] Step 3 also includes:

[0026] Color via spatial rendering :

[0027] ,

[0028] Where M is the number of projected Gaussians corresponding to each rendered pixel, , are the color and opacity of the kth Gaussian point respectively; is the opacity of the kth Gaussian point.

[0029] Step 3 also includes: Obtaining depth through spatial rendering :

[0030] ,

[0031] in From the target camera to the kth Gaussian point distance;

[0032] Normal vector of the surface extracted from depth for:

[0033] ,

[0034] in For Depth spatial location; To find the derivative in the x-axis direction, To find the derivative in the y-axis direction.

[0035] Step 3 also includes:

[0036] Obtaining the surface normal of Gaussian point cloud through spatial rendering :

[0037] ,

[0038] in is the direction of the shortest axis of the Gaussian point, which is determined by the three-dimensional scale of each Gaussian point The direction of the smallest dimension is determined by

[0039] Step 4 includes: using the following formula to calculate the corresponding position of the training view pixel to the neighboring view :

[0040] ,

[0041] in and Represents the training perspective The horizontal and vertical coordinates of and Respectively represent the viewing angle The horizontal and vertical coordinates of

[0042] At the same time, we get the geometric optical flow .

[0043] In step 5, the prior optical flow .

[0044] In step 6, the optical flow loss for:

[0045] .

[0046] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0047] The method of the present invention aims to improve the geometric quality of the three-dimensional Gaussian radiation field at unobserved viewpoints by introducing matching prior information. Specifically, the study found that the matching information between the training viewpoint and the unobserved viewpoint generated by the matching prior model showed higher robustness and accuracy in the early stage of geometric optimization than the matching information directly calculated based on the radiation field geometry. Even when the radiation field geometry has not been fully optimized and the images rendered from the unobserved viewpoint are blurred during the training process, the matching prior can still provide robust geometric guidance.

[0048] To this end, the optical flow generated by the matching prior model is called "prior optical flow", and the optical flow based on the geometric calculation of the radiation field is called "geometric optical flow". During the training process, this method uses the prior optical flow to optimize the geometric optical flow, thereby improving the rendering quality and geometric performance of the radiation field under unobserved perspectives. Specifically, the prior optical flow provides a reliable initial geometric guidance, making the radiation field more stable in the optimization of unobserved perspectives. As the training progresses, the geometric information of the radiation field is gradually optimized, and the image rendered under adjacent perspectives becomes clearer, thereby improving the accuracy of the prior optical flow. In this process, the prior optical flow and the geometric optical flow reinforce each other, forming a gradually optimized positive feedback mechanism. In this way, the geometric and radiation information of the radiation field can be effectively improved under unobserved perspectives, further improving the overall reconstruction accuracy.

[0049] This innovative online distillation method not only solves the problem of geometric instability of radiation fields in sparse viewports, but also significantly improves the quality of images rendered from new viewports by co-optimizing between matching priors and geometric information.

[0050] The advantages and beneficial effects of the present invention include:

[0051] 1. Improvement of geometric accuracy: This invention significantly improves the geometric reconstruction accuracy of Gaussian radiation field by using the optical flow model matching information of unobserved viewpoints during training. Compared with other geometric prior information, the matching flow prior information combined with the pose input can restore the absolute scale of the scene, thus providing more reliable guidance for geometric optimization. This solves the scale ambiguity problem commonly seen in monocular geometry inference, allowing the radiation field to obtain a more accurate geometric representation in complex scenes.

[0052] 2. Make full use of unobserved viewpoints: This paper introduces information from unobserved viewpoints to fully distill the geometric knowledge of the pre-trained optical flow model, further improving the reconstruction quality of the radiation field. Unobserved viewpoints not only provide additional geometric information, but also promote the coordinated development between geometry and radiation field through joint optimization with training viewpoints.

[0053] 3. Robustness of matching priors: The matching priors generated by the pre-trained optical flow model are more robust than the optical flow based on geometric calculations, especially in the early stages when the radiation field has not been fully optimized. By combining the prior optical flow with the geometric optical flow, this method achieves mutual enhancement of the two during the training process, gradually improving the geometric accuracy and consistency between perspectives.

[0054] In summary, the present invention solves the problem of geometric reconstruction of 3D Gaussian radiation field under sparse perspective by introducing matching prior information of the optical flow model, greatly improves the image quality rendered from an unobserved perspective, and provides an effective solution for high-precision three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0056] Figure 1 Schematic diagram of the method of the present invention.

[0057] Figure 2 It is a schematic diagram comparing the results of the embodiments of the present invention with the prior art. DETAILED DESCRIPTION

[0058] In this embodiment, a Gaussian radiation field 3D reconstruction method based on geometric distillation of a pre-trained optical flow model is provided, comprising the following steps:

[0059] Step 1: User inputs a set of training images And the corresponding pose matrix , the camera's intrinsic parameter matrix K and a pre-trained matching model ; Initialize the Gaussian radiation field at the same time ,in is the i-th Gaussian point, represents the i-th training picture, B represents the total number of training pictures, Represents the pose matrix of the i-th training picture, and the pose matrix is ​​the viewing angle; are the parameters of the pre-trained matching model; the pre-trained matching model is an optical flow model pre-trained on the Sintel dataset commonly used in the academic field; , , , , , They correspond to the rotation, scale, color, position and opacity of the i-th Gaussian point respectively;

[0060] Step 2: Execute the optimization process of Gaussian radiation field: First, randomly select a perspective from the perspective of the training image. As the current training perspective, from this perspective Sampling an unobserved view in the neighborhood of ; The range of the neighborhood is a circle with a radius of r; ; For perspective The average depth of the image, f is the focal length, is a user-defined parameter, which is set to 8.0 in this embodiment;

[0061] Step 3: Rendering perspective Pictures , render the current training perspective Pictures And the corresponding geometric depth map; the expression of the Gaussian radiation sphere is:

[0062] ,

[0063] in is the distribution of the i-th Gaussian radiation sphere at the spatial position X, is the center position of the i-th Gaussian radiation sphere, e is a natural constant, T represents the matrix transpose, By rotation and scale Decide: ; where S is the scale The parameters obtained by diagonalization, R is the quaternion rotation The resulting rotation matrix.

[0064] Color via spatial rendering :

[0065] ,

[0066] Where M is the number of projected Gaussians corresponding to each rendered pixel, , are the color and opacity of the kth Gaussian point respectively; is the opacity of the kth Gaussian point.

[0067] Gaining Depth with Spatial Rendering :

[0068] ,

[0069] in From the target camera to the kth Gaussian point distance;

[0070] Normal vector of the surface extracted from depth for:

[0071] ,

[0072] in For Depth spatial location; To find the derivative in the x-axis direction, To find the derivative in the direction of the y-axis, the x-axis and y-axis here are the world coordinate system.

[0073] Obtaining the surface normal of Gaussian point cloud through spatial rendering :

[0074] ,

[0075] in is the direction of the shortest axis of the Gaussian point, which is determined by the three-dimensional scale of each Gaussian point The direction of the smallest dimension is determined by

[0076] Step 4: Through the current training perspective Depth , and the current training perspective With perspective The pose transformation , calculate the corresponding position of the training view pixel to the neighborhood view :

[0077] ,

[0078] in and Represents the training perspective The horizontal and vertical coordinates of and Respectively represent the viewing angle The horizontal and vertical coordinates of

[0079] At the same time, we get the geometric optical flow .

[0080] Step 5: Change the viewing angle Pictures And the training picture input pre-trained matching model , get the prior optical flow ;

[0081] Step 6: Use prior optical flow To supervise the geometric optical flow , and get the optical flow loss , combine color loss and SSIM structural similarity loss, perform gradient back propagation, and optimize the 3D Gaussian radiation field; return to step 2 until the optimization converges, and the overall optimization loss function L is:

[0082] ;

[0083] in, is the structural similarity loss weight, is the optical flow loss weight, represents the color loss function, represents the structural similarity loss function, is the surface normal consistency loss, is the surface normal consistency loss weight; It can also be replaced with other normals for supervision, such as the surface normal predicted by a pre-trained large model.

[0084] Step 7: After the loss decreases, the optimization ends and the final Gaussian radiation field is obtained.

[0085] Table 1 shows the reconstruction results of the method in this embodiment on the MushRoom dataset. The lower the Acc, Comp and C-L1, the higher the reconstruction quality, where Acc is the accuracy, Comp is the completeness, and C-L1 is the chamfer distance. The higher the NC and F-Score, the higher the reconstruction quality, where NC is the consistency of the surface normal, and F-score is the F score, which comprehensively considers the accuracy and recall rate. PSNR is the root mean square signal-to-noise ratio, and the higher the PSNR, the better the color rendering quality.

[0086] Table 1

[0087] method Acc Comp C-L1 NC F-Score PSNR Gaussian radiation field 0.12 0.12 0.12 0.74 0.26 24.40 Gaussian radiation field + (invention) 0.05 0.07 0.06 0.76 0.63 24.57

[0088] like Figure 2 As shown, the original Gaussian radiation field has an overfitting problem, which leads to the corruption of the geometry generation. Figure 2 As shown in the first row. The Gaussian radiation field combined with the method of the present invention effectively alleviates this problem and helps restore the geometric structure by supervising the matching relationship between the training view and the sampling view. The method of the present invention also improves the fineness of the geometric details, such as Figure 2 As shown in the second row, by introducing the matching prior of the sampled viewpoint during training, the quality of the rendered depth and the surface normal map extracted from the depth correspondence is significantly improved.

[0089] During the training process, the method of the present invention samples unobserved viewpoints from the surrounding of the training viewpoints, and uses the prior optical flow information between these viewpoints and the training viewpoints to guide the geometric optimization of the radiation field. This matching prior not only enhances the geometric consistency between viewpoints, but also effectively reduces the geometric reconstruction error caused by sparse viewpoints.

[0090] By introducing the prior flow information generated by the pre-trained optical flow matching model into the optimization process, the present invention can more accurately constrain the geometric structure of the radiation field, especially when the view angle is sparse or there are complex scene changes. The introduction of the prior flow provides a more reliable reference for geometric optimization, so that the radiation field can maintain high-quality geometric performance even at unobserved view angles.

[0091] Experiments conducted on mesh extraction and new perspective synthesis have verified the effectiveness of this method. Experimental results show that compared with the existing state-of-the-art methods, the geometric matching prior integration method (FDS) proposed in this paper has achieved significant improvements in 3D scene reconstruction and new perspective rendering. Specifically, the reconstructed geometric mesh is more accurate, the surface details are richer, and the visual effect is more natural. In addition, in the new perspective synthesis task, the generated images show higher clarity and perspective consistency, and reduce common artifacts and blurring.

[0092] In summary, the three-dimensional Gaussian radiation field optimization method based on geometric matching prior proposed in the present invention provides a novel and efficient solution for geometric reconstruction and perspective synthesis, which significantly improves the geometric accuracy and rendering quality in complex scenes.

[0093] The present invention provides a Gaussian radiation field 3D reconstruction method based on geometric distillation of a pre-trained optical flow model. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A 3D reconstruction method of Gaussian radiation field based on geometric distillation of a pre-trained optical flow model, characterized in that: The following steps are involved: Step 1: Input training image set And the corresponding pose matrix The camera's intrinsic parameter matrix K and a pre-trained matching model M θ ; Initialize the Gaussian radiation field at the same time Where P i is the i-th Gaussian point, I i represents the i-th training picture, B represents the total number of training pictures, T i Represents the pose matrix of the i-th training picture, and the pose matrix is ​​the viewing angle; θ is the parameter of the pre-trained matching model; the pre-trained matching model is an optical flow model; P i = {r i ,s i ,c i , i ,α i }, r i ,s i ,c i ,x i ,α i They correspond to the rotation, scale, color, position and opacity of the i-th Gaussian point respectively; Step 2: Execute the optimization process of Gaussian radiation field: First, randomly select a view angle T from the view angles of the training image. i As the current training perspective, from this perspective T i Sample an unobserved view T in the neighborhood of s ; The range of the neighborhood is a circle with a radius of r; Step 3: Rendering View T s Image of C s , render the current training view T i Image of C i And the corresponding geometric depth map D i ; Step 4: Through the current training perspective T i The depth D(X) and the current training perspective T i With viewing angle T s The pose transformation Calculate the corresponding position of the training view pixel to the neighborhood view to obtain the geometric optical flow Step 5: Change the viewing angle T s Image of C s And the training picture input pre-trained matching model M θ , get the prior optical flow Step 6: Use prior optical flow To supervise the geometric optical flow Get the optical flow loss L fds , combine color loss and structural similarity loss, perform gradient back propagation, and optimize the 3D Gaussian radiation field; return to step 2 until the optimization converges, and the overall optimization loss function L is: L=(1-λ)L1(C i ,THE i )+λL D-SSIM (C i ,THE i )+λ fds THE fds +λ d THE N (N d (X),N(X)); Where λ is the structural similarity loss weight, λ fds is the optical flow loss weight, L1 represents the color loss function, L D-SSIM represents the structural similarity loss function, L N is the surface normal consistency loss, λ d is the surface normal consistency loss weight; N d (X) represents the normal vector of the depth-extracted surface, and D(X) is the depth obtained by spatial rendering; Step 7: After the loss decreases, the optimization ends and the final Gaussian radiation field is obtained.

2. The method according to claim 1, characterized in that In step 2, D i The viewing angle T i is the average depth of , f is the focal length, and ε is a user-defined parameter.

3. The method according to claim 2, characterized in that Step 3 includes: The expression of Gaussian radiation sphere is: Among them G i (X) is the distribution of the i-th Gaussian radiation ball at the spatial position X, μ i is the center position of the i-th Gaussian radiation sphere, e is a natural constant, T represents the matrix transpose, Σ i By rotating r i and scales i Decision: Σ i =RSS T R T ; where S is the scale s i The parameters obtained by diagonalization, R is the quaternion rotation r i The resulting rotation matrix.

4. The method according to claim 3, characterized in that Step 3 also includes: Obtain color C(X) through spatial rendering: Where M is the number of Gaussians after projection corresponding to each rendered pixel, c k , α k are the color and opacity of the kth Gaussian point respectively; α j is the opacity of the kth Gaussian point.

5. The method according to claim 4, characterized in that Step 3 also includes: obtaining the depth D(X) by spatial rendering: The normal vector N of the surface extracted from depth d (X) is: where d k From the target camera to the kth Gaussian point P k The distance; p s is the spatial position of the depth E(X); To find the derivative in the direction of the x-axis, To find the derivative in the y-axis direction.

6. The method according to claim 5, characterized in that Step 3 also includes: Obtain the Gaussian point cloud surface normal N(X) through spatial rendering: where n k is the direction of the shortest axis of the Gaussian point, which is determined by the three-dimensional scale s of each Gaussian point i The direction of the smallest dimension is determined by 7. The method according to claim 6, characterized in that Step 4 includes: using the following formula to calculate the corresponding position D(u s ,v s )[u s ,v s ,1]: where u i and v i Represent the training view T i The horizontal and vertical coordinates of u s and v s Respectively represent the viewing angle T s The horizontal and vertical coordinates of At the same time, we get the geometric optical flow 8. The method according to claim 7, characterized in that In step 5, the prior optical flow 9. The method according to claim 8, characterized in that In step 6, the optical flow loss L fds for:

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.

Images