Sparse input scene reconstruction method based on 3D Gaussian sputtering and computer program product

Abstract

The invention provides a sparse input scene reconstruction method based on 3D Gaussian sputtering and a computer program product, and the method comprises the steps: inputting 2D images of different visual angles of the same scene into colmap software, and obtaining a 3D Gaussian initialization point cloud, the initialization point cloud is used for determining the number and attributes of Gaussian points at the beginning of scene reconstruction; the measurement distance between a Gaussian point A and a Gaussian point B is obtained, the Gaussian point A is any one Gaussian point, and the Gaussian point B is a Gaussian point closest to the Gaussian point A; if the measurement distance is greater than a first threshold value, adding a Gaussian point C between the Gaussian point A and the Gaussian point B; and standardizing 3D Gaussian scene reconstruction based on the 2D image to obtain a reconstructed three-dimensional scene. According to the method, after the Gaussian point C is supplemented and added, the initialized point cloud is perfected, the dependence on a large number of input pictures during three-dimensional scene reconstruction is reduced, and the reconstructed three-dimensional scene keeps better richness in the three-dimensional scene reconstruction optimization process.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a sparse input scene reconstruction method based on 3D Gaussian sputtering and a computer program product. Background Art

[0002] Sparse input scene reconstruction is an important task in the field of vision and graphics. It can cope with the situation of few input images and reconstruct the scene with higher quality under sparse input conditions. Given a small number of pictures from different perspectives, due to the lack of sufficient 3D information, directly using traditional methods often produces poor rendering results, which is usually manifested as overfitting of the scene to the input perspective image and poor synthesis results of new perspectives. In order to improve the reconstruction quality of sparse input scenes, existing methods have explored a variety of methods to better utilize input information for constraints. However, they still have problems such as overfitting of the input image.

[0003] Neural Radiance Fields (NeRF) proposed to use neural networks to model 3D scenes. Due to its excellent performance in reconstructing geometry and appearance, it has become an excellent scene representation. Due to its reliability, there are currently many methods that use it to reconstruct sparse input scenes. DietNeRF (Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis) introduces a trained ViT (Vision Transformer) model to ensure that the semantic information of images under unknown viewpoints is the same as that of the reference image, thereby preventing the degradation phenomenon during scene reconstruction. FreeNeRF (FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization) further improves the quality of 3D reconstruction by enhancing the scene's perception of low-frequency signals. However, despite the valuable efforts, due to the limitations of NeRF as a representation, these works have a lot of room for improvement in terms of training time and rendering time.

[0004] Recently, the introduction of 3D Gaussian Splatting (3DGS) provides a new idea for the reconstruction of sparse input scenes. Compared with NeRF, 3DGS not only improves the reconstruction quality of three-dimensional scenes, but also greatly speeds up the training and rendering speeds. Therefore, how to use 3DGS to reconstruct sparse input scenes has become one of the research hotspots. Since 3DGS needs to use the point cloud output by colmap when initializing, there are too few Gaussian points in the scene when initializing under sparse input conditions, which is not conducive to subsequent optimization. Summary of the invention

[0005] Based on this, it is necessary to provide a sparse input scene reconstruction method based on 3D Gaussian sputtering to address the above technical problems.

[0006] The present application provides a sparse input scene reconstruction method based on 3D Gaussian sputtering, including:

[0007] Input 2D images of the same scene from different perspectives into the colmap software to obtain a 3D Gaussian initialization point cloud, which is used to determine the number and properties of Gaussian points at the beginning of scene reconstruction;

[0008] Obtaining a measured distance between a Gaussian point A and a Gaussian point B, wherein the Gaussian point A is any Gaussian point, and the Gaussian point B is the Gaussian point closest to the Gaussian point A;

[0009] If the measured distance is greater than a first threshold, adding a Gaussian point C between Gaussian point A and Gaussian point B;

[0010] The scene is reconstructed by normalizing the 3D Gaussian based on the 2D image to obtain a reconstructed three-dimensional scene.

[0011] Optionally, standardizing the scene reconstruction of the 3D Gaussian based on the 2D image includes: using the 2D image and the rendered image of the 3D Gaussian at a corresponding angle, and an average value of the absolute difference between the two, to standardize the scene reconstruction.

[0012] Optionally, including: reconstructing the scene of the 3D Gaussian using depth loss specification, specifically including:

[0013] Performing Gaussian rendering on the 3D image to obtain a first depth map and an RGB image, and performing depth estimation on the RGB image to obtain a second depth map;

[0014] Get the first depth map and the second depth map Figure 2 The depth loss of the 3D Gaussian is optimized using the depth loss.

[0015] Optionally, the depth loss is obtained by:

[0016]

[0017] represents the depth loss;

[0018] E, means seeking expectation;

[0019] D r , pixel r of the first depth map;

[0020] D rh , the upper pixel of the pixel r of the first depth map;

[0021] D′ r , the pixel point r at the corresponding position of the second depth map;

[0022] D′ rh , the upper pixel point of the pixel point r at the corresponding position in the second depth map.

[0023] Optionally, if the measured distance is less than a second threshold, the Gaussian point A is deleted.

[0024] Optionally, a Gauss point C is added between Gauss point A and Gauss point B, specifically including:

[0025] The Gaussian point A has a first distance from the Gaussian point C, the Gaussian point B has a second distance from the Gaussian point C, and the position of the Gaussian point C is determined by making the ratio of the second distance to the first distance equal to the ratio of the importance value scores of the Gaussian point A to the Gaussian point B;

[0026] The importance score is the maximum value of the sum of the volume rendering weights of a Gaussian point under all rendering perspectives, and is evaluated based on the repetition of use of the Gaussian point when rendering at different perspectives.

[0027] Optionally, the position of the Gauss point C is determined by:

[0028]

[0029] Where:

[0030] μ C , represents the position of the added Gaussian point C;

[0031] μ A , represents the position of Gauss point A;

[0032] Score A , represents the important value score of Gaussian point A;

[0033] μ B , represents the position of Gauss point B;

[0034] Score B, represents the important value score of Gaussian point B;

[0035] The importance score is obtained using the following formula:

[0036]

[0037] Where:

[0038] Score i , represents the important value score of the i-th Gaussian point;

[0039] I, represents all rendering perspectives;

[0040] j, represents one of the rendering perspectives I;

[0041] r, represents the ray at the rendering perspective j;

[0042]

[0043] Indicates the opacity corresponding to the Gaussian point in the direction of ray r;

[0044] Represents the distance between adjacent sampling points of the Gaussian point in the direction of ray r.

[0045] Optionally, including: using Gaussian point spacing loss to normalize the 3D Gaussian scene reconstruction, the Gaussian point spacing loss is obtained using the following formula:

[0046]

[0047] L d2 , represents the Gaussian point spacing loss;

[0048] i, j, represent any two Gaussian points;

[0049] w i , represents the volume rendering weight;

[0050] z j , represents the distance from the Gaussian point to the imaging plane.

[0051] Optional, including: using virtual perspective to standardize scene reconstruction, specifically including:

[0052] In a three-dimensional space, obtaining any two first camera perspectives and second camera perspectives corresponding to the 2D image;

[0053] Inserting a virtual camera between the positions of the first camera angle of view and the second camera angle of view, and randomly setting the virtual camera angle of view for the virtual camera;

[0054] The virtual camera perspectives and the 2D image semantic information are sorted and screened in terms of similarity, and the virtual camera perspectives with the highest similarity rankings are taken and combined with the known perspectives of the 2D image to standardize the scene reconstruction.

[0055] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the sparse input scene reconstruction method based on 3D Gaussian sputtering described in the present application.

[0056] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the sparse input scene reconstruction method based on 3D Gaussian sputtering described in the present application.

[0057] The present application also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the sparse input scene reconstruction method based on 3D Gaussian sputtering described in the present application.

[0058] The sparse input scene reconstruction method based on 3D Gaussian sputtering in this application has at least the following effects:

[0059] After the application adds Gauss point C between Gauss point A and Gauss point B, the initialization point cloud is improved, and the 3D scene reconstruction can be performed even in the case of sparse input with insufficient number of 2D images. The application reduces the dependence on a large number of input images when reconstructing the 3D scene, and in the subsequent 3D scene reconstruction optimization process, the reconstructed 3D scene maintains a good richness. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flowchart of a sparse input scene reconstruction method based on 3D Gaussian sputtering in one embodiment of the present application;

[0061] Figure 2 A schematic diagram of a process of deleting and adding Gaussian points in a sparse input scene reconstruction method based on 3D Gaussian sputtering in an embodiment of the present application;

[0062] Figure 3 Schematic diagram of a model framework of a sparse input scene reconstruction method based on 3D Gaussian sputtering in one embodiment of the present application;

[0063] Figure 4 Schematic diagram of a model framework of a sparse input scene reconstruction method based on 3D Gaussian sputtering in one embodiment of the present application;

[0064] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] Each embodiment of the present application mainly adopts a three-dimensional explicit expression model based on 3DGS, that is, based on the input image, that is, based on the three-dimensional scene reconstructed by the input 2D image, 3D Gaussian points in space are used for accurate geometric modeling and visual rendering. 3DGS projects the Gaussian points in space onto a 2D plane according to the viewing direction, and obtains the final image through volume rendering. However, 3DGS uses the point cloud generated by the colmap software based on the input image as the initialization condition, and the Gaussian points added during the optimization process are all close to the initial points. Therefore, when the input image is too small, the number of point clouds generated by the software is too small. In the reconstruction process of the three-dimensional scene, although new Gaussian points can be automatically generated, the automatically generated new Gaussian points present a situation of surrounding the Gaussian points in the initialization point cloud, which is not conducive to the subsequent reconstruction and optimization of the three-dimensional scene.

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0067] See also Figure 1 to Figure 3 An embodiment of the present application provides a sparse input scene reconstruction method based on 3D Gaussian sputtering, including steps S100 to S400. Among them:

[0068] Step S100, inputting 2D images of the same scene from different perspectives into the colmap software to obtain an initialization point cloud of 3D Gaussian points, which is used to determine the number and properties of Gaussian points at the beginning of scene reconstruction.

[0069] Step S200 , obtaining the measured distance between Gaussian point A and Gaussian point B, where Gaussian point A is any Gaussian point, and Gaussian point B is the Gaussian point closest to Gaussian point A.

[0070] Step S300: if the measured distance is greater than the first threshold, add a Gaussian point C between Gaussian point A and Gaussian point B.

[0071] Step S400, reconstructing a scene based on a 3D Gaussian of a 2D image to obtain a reconstructed three-dimensional scene.

[0072] For step S100, Figure 3 SfM, structure from motion, means recovering structure from motion. Figure 3As shown, step S400 includes using a 2D image and a rendered image of a 3D Gaussian corresponding angle (specifically, an RGB image obtained by rendering), and the average value of the absolute difference between the two to normalize the scene reconstruction. When the number of input 2D images is insufficient, the number of Gaussian points included in the initialization point cloud is insufficient, which affects the subsequent reconstruction optimization process. In this embodiment, steps S200 to S300 are performed repeatedly as a way to densify spatial Gaussian points. In this embodiment, since Gaussian point C is added by adapting to Gaussian point A and Gaussian point B, after the Gaussian point C is added, the initialization point cloud is improved, and it is competent for three-dimensional scene reconstruction even in the case of sparse input with insufficient number of 2D images. This embodiment reduces the dependence on a large number of input images during three-dimensional scene reconstruction, and in the subsequent three-dimensional scene reconstruction optimization process, the reconstructed three-dimensional scene maintains a good richness.

[0073] In step S200, the measured distance between Gaussian point A and Gaussian point B is obtained by the following formula:

[0074]

[0075] Formula (1) calculates the distance between each Gaussian point and its nearest Gaussian point as the basis for the densification of spatial Gaussian points.

[0076] D i , represents the distance between a Gaussian point and its nearest Gaussian point, that is, the measured distance between Gaussian point A and Gaussian point B mentioned above;

[0077] i and j are both used to represent a Gaussian point;

[0078] d ij , represents the distance between the target Gaussian point and other Gaussian points;

[0079] μ represents the location of the Gaussian point.

[0080] Step S300: If the measured distance is greater than the first threshold, a Gaussian point C is added between Gaussian point A and Gaussian point B. Specifically, when the measured distance exceeds the preset first threshold, a new Gaussian point C is set between Gaussian point A and Gaussian point B, corresponding to Figure 2 , click Add this Gaussian point.

[0081] Furthermore, if the measured distance is less than the second threshold, the Gaussian point A is deleted, corresponding to Figure 2 In the above equation, "delete the Gaussian point". When the distance between a Gaussian point and its nearest Gaussian point is too large, delete the Gaussian point to remove the noise. The above process is expressed by equation (2):

[0082] m i =τ add <Di <τ prune (2)

[0083] m i , represents a threshold-based binary mask used to select Gaussian points involved in densification;

[0084] τ add , represents the first threshold, if D i Greater than the first threshold τ add , then add a Gaussian point, that is, add a Gaussian point C between Gaussian point A and Gaussian point B.

[0085] τ prune , represents the second threshold, if D i Greater than or equal to the second threshold τ prune , then remove the Gaussian point.

[0086] Formula (2) determines the Gaussian points that need to participate in the 3D Gaussian densification process. i Greater than τ prune When , the Gauss point is pruned. The attributes of the new Gauss point are set according to the importance score of the related Gauss point. The attributes of the new Gauss point, that is, the attributes of the Gauss point C, include other attributes such as position attributes and color. The importance score of each Gauss point is calculated during the rendering process.

[0087] In step S300, adding a Gaussian point C between Gaussian point A and Gaussian point B specifically includes:

[0088] Gaussian point A has a first distance from Gaussian point C, Gaussian point B has a second distance from Gaussian point C, and the position of Gaussian point C is determined by making the ratio of the second distance to the first distance equal to the ratio of the importance value scores of Gaussian point A and Gaussian point B. The importance value score is the maximum value of the sum of the volume rendering weights of a Gaussian point under all rendering perspectives, and is evaluated based on the use repetition of the Gaussian point when rendering at different perspectives.

[0089]

[0090] In the formula, μ new Indicates the position of the new Gaussian point in space; i and j are both used to represent a Gaussian point; Score i 、Score j Represent the important value scores of the i-th and j-th Gaussian points respectively.

[0091] Specifically, after obtaining the importance value score of each Gaussian point, the position attribute of Gaussian point C is set according to the score relationship between Gaussian point A and its nearest Gaussian point B that satisfies equation (3). The position of Gaussian point C is determined by:

[0092]

[0093] In the formula,

[0094] μ C , represents the position of the added Gaussian point C;

[0095] μ A , represents the position of Gauss point A;

[0096] Score A , represents the important value score of Gaussian point A;

[0097] μ B , represents the position of Gauss point B;

[0098] Score B , represents the important value score of Gaussian point B.

[0099] The above formula can be used to determine the position attribute of Gauss point C in space. At the same time, the remaining attributes of Gauss point C are set to be the same as those of the Gauss point with a larger importance value score. That is, for the newly added Gauss point C, after confirming the position attribute of Gauss point C, the remaining attributes of Gauss point C are set to be equal to one of Gauss point A and Gauss point B that is adjacent to Gauss point C.

[0100] This scheme places the position of Gaussian point C close to the one with a larger importance value score between Gaussian point A and Gaussian point B, so that the newly added Gaussian point can make up for the deficiency of the input initialization point cloud. In the case of sparse input, the importance score can reflect the lack of Gaussian points in the scene to a certain extent. This scheme uses the importance score to reasonably confirm the position of the newly added Gaussian points and adaptively densifies the spatial Gaussian points to achieve higher-quality reconstruction. This scheme avoids the increase of invalid Gaussian points in space through adaptive judgment based on the importance score, effectively suppresses overfitting, and further improves the quality value of its reconstructed scene, which is helpful for the subsequent 3D Gaussian reconstruction optimization.

[0101] Due to the newly added position attribute of Gauss point C, it is set according to the importance score of the related Gauss point, and the importance score of each Gauss point is calculated during the rendering process.

[0102]

[0103] Where:

[0104] r, represents the ray under the rendering perspective;

[0105] Indicates the ray color at the rendering perspective;

[0106] Indicates the color of the Gaussian point;

[0107]

[0108] σ i , represents the opacity corresponding to the Gaussian point in space;

[0109] δ i , represents the distance between adjacent sampling points in the ray direction, δ i =t i+1 -t i , t i is the sampling point, used to represent the sampling of Gaussian points.

[0110] According to equation (3), the importance of each Gaussian point in rendering at different views can be intuitively used as the weight “T i (1-exp(-σ i δ i )”, in this way, the important value score can be obtained. This scheme uses the maximum value of the sum of the volume rendering weights of the Gaussian point under all viewing angles as the important value score of the Gaussian point, and the important value score is obtained using the following formula:

[0111]

[0112] Where:

[0113] Score i , represents the important value score of the i-th Gaussian point;

[0114] I, represents all rendering perspectives;

[0115] j, represents one of the rendering perspectives I;

[0116] r, represents the ray at the rendering perspective j;

[0117]

[0118] Indicates the opacity corresponding to the Gaussian point in the direction of ray r;

[0119] Represents the distance between adjacent sampling points of the Gaussian point in the direction of ray r.

[0120] See also Figure 4 In some embodiments, the sparse input scene reconstruction method based on 3D Gaussian sputtering includes step S500, reconstructing the scene using a depth loss-normalized 3D Gaussian.

[0121] While using 3D Gaussian to perform reconstruction optimization, this embodiment uses depth constraints to further standardize the geometric structure of the scene to prevent Gaussian points from being located at incorrect positions.

[0122] As a depth constraint, step S500 uses depth loss to normalize the scene reconstruction of the 3D Gaussian, specifically including: rendering the 3D Gaussian to obtain a first depth map and an RGB image, performing depth estimation on the RGB image to obtain a second depth map; obtaining the first depth map and the second depth map Figure 2 The depth loss of the original image is used to optimize the 3D Gaussian.

[0123] Depth estimation of RGB images can be implemented using a pre-trained monocular depth estimation model. The depth map estimated by the pre-trained monocular depth estimation model can represent the relative relationship of depth. This embodiment implements depth constraints through the relative relationship between pixels in the two depth maps, and abandons directly using the difference between the depth and the rendered depth as the loss function. The depth loss is obtained by:

[0124]

[0125] Where:

[0126] represents the depth loss;

[0127] E, means seeking expectation;

[0128] D r , pixel r of the first depth map;

[0129] D rh , the upper pixel of the pixel r of the first depth map;

[0130] D′ r , the pixel point r at the corresponding position of the second depth map;

[0131] D′ rh , the upper pixel point of the pixel point r at the corresponding position in the second depth map.

[0132] Formula (6) calculates the difference between each corresponding position pixel and the upper pixel in the first depth map and the second depth map, and only when the difference signs are different, the absolute value of the multiplication of the two is included in the loss function to improve the reliability of the depth loss.

[0133] In some embodiments, the depth loss includes and The sum of the two parts, formula (6) explains how to calculate the loss function by the difference with the upper pixel value method, It means that the loss function is calculated by the difference with the pixel value on the right. The calculation process is similar to that of formula (6), so that the depth information is finally used to constrain the geometric reconstruction process of the scene.

[0134] See also Figure 4 In some embodiments, the sparse input scene reconstruction method based on 3D Gaussian sputtering includes step S600, using Gaussian point spacing loss to standardize the 3D Gaussian scene reconstruction, and concentrating the Gaussian points on the scene surface through the constraint of Gaussian point spacing loss to prevent the degradation of the rendered image quality under unknown viewing angles.

[0135] The Gaussian point spacing loss is obtained using the following formula:

[0136] L d2 =∑ i，j w i w j |z i -z j | (7)

[0137] L d2 , is the regularization term of Gaussian points, indicating the Gaussian point spacing loss;

[0138] i, j, represent any two Gaussian points;

[0139] w i , represents the volume rendering weight, that is, the weight "T i (1-exp(-σ i δ i )";

[0140] z j , represents the distance from the Gaussian point to the imaging plane.

[0141] Equation (7) makes the Gaussian points as close to the surface of the object as possible, effectively alleviating the problem of incorrect Gaussian point positions caused by sparse input.

[0142] In some embodiments, the sparse input scene reconstruction method based on 3D Gaussian sputtering includes step S700, using virtual perspective to standardize scene reconstruction. As a pseudo perspective synthesis constraint, step S700 specifically includes: in three-dimensional space, obtaining any two first camera perspectives and second camera perspectives corresponding to 2D images; inserting a virtual camera between the positions of the first camera perspective and the second camera perspective, and randomly setting the virtual camera perspective for the virtual camera; sorting and screening the virtual camera perspective and the 2D image semantic information progress similarity, taking the virtual camera perspectives ranked at the top of the similarity, and combining them with the known perspective of the 2D image to standardize the scene reconstruction.

[0143] This embodiment can further prevent the occurrence of overfitting and use the pseudo-perspective (virtual perspective) image to further standardize the reconstruction of the scene.

[0144] The perspectives are sampled from any two training views (input 2D images) of (three-dimensional space), and a virtual camera is interpolated between them, and the camera direction is randomly set. Since there is no prior knowledge under pseudo-perspective, the pre-trained ViT (Vision Transformer) model can be used to screen perspectives with similar semantic information to the reference perspective, and the depth constraint method similar to step S500 is used under these perspectives to optimize the reconstruction quality of the three-dimensional scene.

[0145] S j =φ(I1) T φ(I j )+φ(I2) T φ(I j ) (8)

[0146] S j , represents the similarity score of pseudo-perspective;

[0147] I1, represents the rendered image from the first camera’s perspective;

[0148] I2, represents the rendered image from the second camera’s perspective;

[0149] I j , represents the rendered image from the perspective of the virtual camera;

[0150] φ, represents the ViT model;

[0151] T, represents matrix transpose.

[0152] In equation (8), I1 and I2 represent the rendered images under the two known viewpoints closest to the pseudo-view camera position, and φ represents the image processing using ViT. After calculating the similarity score of each pseudo-viewpoint, the scores are sorted in ascending order, and the top 50% of pseudo-viewpoints are selected to combine with the known viewpoints and provide geometric constraints for scene reconstruction.

[0153] When the steps provided in the embodiments of the present application are used simultaneously, the overall loss is L total .

[0154] L total =L1+L d1 +L d2 (9)

[0155] in,

[0156] L total , indicating the overall loss;

[0157] L1, represents the average of the absolute differences between the 2D image and the 3D Gaussian corresponding angle rendered image;

[0158] Ld1 , represents the depth loss,

[0159] L d2 , represents the Gaussian point spacing loss.

[0160] Equation (9) can be used as the loss function of the entire optimization process, thereby achieving 3D scene reconstruction under sparse input conditions.

[0161] In each embodiment of the present application, a sparse input scene reconstruction method based on 3D Gaussian sputtering is provided and a framework as shown in the accompanying drawings is constructed accordingly. The sparse input scene reconstruction method can not only reconstruct the scene in the case of sparse input well, but also can adaptively adjust the number of Gaussian points in the space. The key insight is to introduce important value scores in the process of densification, combine depth constraints with unknown perspective (pseudo-perspective) constraints, and effectively prevent overfitting as a whole. Among them, step S200 and step S300, as a way of adaptive densification, can adaptively increase the number of Gaussian points in the sparse input three-dimensional scene. Step S500, as a depth constraint, uses a depth map to constrain the scene, which can improve the quality of the geometric structure reconstruction of the scene. Step S700, as a pseudo-perspective synthesis constraint, can synthesize pseudo-perspective images to provide more input for three-dimensional scene reconstruction.

[0162] It should be understood that although Figure 1 The steps in the flowchart are displayed in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps S100 to S700 may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0163] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a sparse input scene reconstruction method based on 3D Gaussian sputtering is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0164] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0165] Step S100, inputting 2D images of the same scene from different perspectives into the colmap software to obtain an initialization point cloud of 3D Gaussian points, which is used to determine the number and properties of Gaussian points at the beginning of scene reconstruction.

[0166] Step S200 , obtaining the measured distance between Gaussian point A and Gaussian point B, where Gaussian point A is any Gaussian point, and Gaussian point B is the Gaussian point closest to Gaussian point A.

[0167] Step S300: if the measured distance is greater than the first threshold, add a Gaussian point C between Gaussian point A and Gaussian point B.

[0168] Step S400, reconstructing a scene based on a 3D Gaussian of a 2D image to obtain a reconstructed three-dimensional scene.

[0169] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0170] Step S100, inputting 2D images of the same scene from different perspectives into the colmap software to obtain an initialization point cloud of 3D Gaussian points, which is used to determine the number and properties of Gaussian points at the beginning of scene reconstruction.

[0171] Step S200 , obtaining the measured distance between Gaussian point A and Gaussian point B, where Gaussian point A is any Gaussian point, and Gaussian point B is the Gaussian point closest to Gaussian point A.

[0172] Step S300: if the measured distance is greater than the first threshold, add a Gaussian point C between Gaussian point A and Gaussian point B.

[0173] Step S400, reconstructing a scene based on a 3D Gaussian of a 2D image to obtain a reconstructed three-dimensional scene.

[0174] In one embodiment, a computer program product is provided, comprising computer instructions, which when executed by a processor implement the following steps:

[0175] Step S100, inputting 2D images of the same scene from different perspectives into the colmap software to obtain an initialization point cloud of 3D Gaussian points, which is used to determine the number and properties of Gaussian points at the beginning of scene reconstruction.

[0176] Step S200 , obtaining the measured distance between Gaussian point A and Gaussian point B, where Gaussian point A is any Gaussian point, and Gaussian point B is the Gaussian point closest to Gaussian point A.

[0177] Step S300: if the measured distance is greater than the first threshold, add a Gaussian point C between Gaussian point A and Gaussian point B.

[0178] Step S400, reconstructing a scene based on a 3D Gaussian of a 2D image to obtain a reconstructed three-dimensional scene.

[0179] In the present embodiment, the computer program product includes a program code portion for executing the steps of the sparse input scene reconstruction method based on 3D Gaussian sputtering in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. The computer program product may also be provided for download via a data network (e.g., via a RAN, via the Internet and / or via an RBS). Alternatively or additionally, the method may be encoded in a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), or functionality may be provided for download by means of a hardware description language.

[0180] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0181] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. When the technical features in different embodiments are embodied in the same figure, it can be regarded that the figure also discloses the combination examples of the various embodiments involved.

[0182] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A sparse input scene reconstruction method based on 3D Gaussian sputtering, characterized in that: include: Input 2D images of the same scene from different perspectives into the colmap software to obtain a 3D Gaussian initialization point cloud, which is used to determine the number and properties of Gaussian points at the beginning of scene reconstruction; Obtaining a measured distance between a Gaussian point A and a Gaussian point B, wherein the Gaussian point A is any Gaussian point, and the Gaussian point B is the Gaussian point closest to the Gaussian point A; If the measured distance is greater than a first threshold, adding a Gaussian point C between Gaussian point A and Gaussian point B; The scene is reconstructed by normalizing the 3D Gaussian based on the 2D image to obtain a reconstructed three-dimensional scene.

2. The sparse input scene reconstruction method according to claim 1, characterized in that: The scene reconstruction of the 3D Gaussian is standardized based on the 2D image, including: using the 2D image and the rendering image of the 3D Gaussian corresponding angle, and the average value of the absolute difference between the two, to standardize the scene reconstruction.

3. The sparse input scene reconstruction method according to claim 1, characterized in that: include: The scene reconstruction of the 3D Gaussian is normalized using the depth loss, specifically including: Performing Gaussian rendering on the 3D image to obtain a first depth map and an RGB image, and performing depth estimation on the RGB image to obtain a second depth map; Depth losses of both the first depth map and the second depth map are obtained, and the 3D Gaussian is optimized using the depth losses.

4. The sparse input scene reconstruction method according to claim 3, characterized in that: The depth loss is obtained by: represents the depth loss; E, means seeking expectation; D r , pixel r of the first depth map; D rh , the upper pixel of the pixel r of the first depth map; D′ r , the pixel point r at the corresponding position of the second depth map; D′ rh , the upper pixel point of the pixel point r at the corresponding position in the second depth map.

5. The sparse input scene reconstruction method according to claim 1, characterized in that: If the measured distance is smaller than the second threshold, the Gaussian point A is deleted.

6. The sparse input scene reconstruction method according to claim 1, characterized in that: Add Gauss point C between Gauss point A and Gauss point B, including: The Gaussian point A has a first distance from the Gaussian point C, the Gaussian point B has a second distance from the Gaussian point C, and the position of the Gaussian point C is determined by making the ratio of the second distance to the first distance equal to the ratio of the importance value scores of the Gaussian point A to the Gaussian point B; The importance score is the maximum value of the sum of the volume rendering weights of a Gaussian point under all rendering perspectives, and is evaluated based on the repetition of use of the Gaussian point when rendering at different perspectives.

7. The sparse input scene reconstruction method according to claim 6, characterized in that: The position of the Gauss point C is determined by: Where: μ C , represents the position of the added Gaussian point C; μ A , represents the position of Gauss point A; Score A , represents the important value score of Gaussian point A; μ B , represents the position of Gauss point B; Score B , represents the important value score of Gaussian point B; The importance score is obtained using the following formula: Where: Score i , represents the important value score of the i-th Gaussian point; I, represents all rendering perspectives; j, represents one of the rendering perspectives I; r, represents the ray at the rendering perspective j; Indicates the opacity corresponding to the Gaussian point in the direction of ray r; Represents the distance between adjacent sampling points of the Gaussian point in the direction of ray r.

8. The sparse input scene reconstruction method according to claim 1, characterized in that: include: The 3D Gaussian scene reconstruction is normalized using Gaussian point spacing loss, which is obtained using the following formula: L d2 , represents the Gaussian point spacing loss; i, j, represent any two Gaussian points; w i , represents the volume rendering weight; z j , represents the distance from the Gaussian point to the imaging plane.

9. The sparse input scene reconstruction method according to claim 1, characterized in that: include: Standardize scene reconstruction using virtual perspective, including: In a three-dimensional space, obtaining any two first camera perspectives and second camera perspectives corresponding to the 2D image; Inserting a virtual camera between the positions of the first camera angle of view and the second camera angle of view, and randomly setting the virtual camera angle of view for the virtual camera; The virtual camera perspectives and the 2D image semantic information are sorted and screened in terms of similarity, and the virtual camera perspectives with the highest similarity rankings are taken and combined with the known perspectives of the 2D image to standardize the scene reconstruction.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the sparse input scene reconstruction method based on 3D Gaussian sputtering as described in any one of claims 1 to 7 are implemented.

Images