Three-dimensional scene and object reconstruction method, system and equipment based on SDF and Gaussian field

Abstract

The invention discloses a three-dimensional scene and object reconstruction method, system and device based on SDF and a Gaussian field, belongs to three-dimensional scene reconstruction in the technical field of computer vision, and aims to solve the technical problem that the three-dimensional scene reconstruction quality is not high due to the fact that geometric precision and rendering quality cannot be achieved at the same time in the three-dimensional reconstruction process in the prior art. The method comprises the following steps: initializing and updating a TSDF voxel volume by using a depth map and an RGB image to obtain a camera external parameter and a scene three-dimensional initial model; sDF prediction is carried out on the scene three-dimensional initial model through an SDF neural network, scene surface points are generated according to the light direction, and a high-precision scene geometric model is obtained; training the SDF neural network; performing grid division on a bounding box of the scene three-dimensional initial model, outputting an SDF predicted value of each grid point by using the trained SDF neural network, and extracting a refined scene grid model; and performing Gaussian rendering by using the high-precision scene geometric model to generate a final rendered image.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, relates to the reconstruction of a three-dimensional scene, and particularly relates to a method, system, and device for reconstructing a three-dimensional scene and an object based on SDF and Gaussian fields. Background Art

[0002] In recent years, three-dimensional scene reconstruction technology has developed rapidly in the fields of computer vision and graphics, and has shown broad application potential in the fields of virtual reality, augmented reality, film production, game development, architectural design, etc. The most basic purpose of scene reconstruction is to recover the geometric structure of a complete three-dimensional scene from two-dimensional images and other sensor data. Among them, the accurate geometric structure of the scene and the rendering quality of new viewpoints are the keys to enhancing the user experience in the above application fields.

[0003] In order to achieve a more realistic three-dimensional scene rendering effect, it is necessary to reconstruct a three-dimensional scene model with a high-precision geometric structure. However, scene reconstruction does face some significant challenges in terms of accurate geometric structure and new viewpoint rendering. The existing methods for scene reconstruction mainly include the following aspects: Scene reconstruction based on SDF (Signed Distance Field): That is, starting from the pixels of a two-dimensional image, emit rays and sample points, input the coordinates of the sample points into the SDF MLP network, use this network to predict the SDF value of the sample points, use the Eikonal equation as the constraint of SDF and RGB as the color supervision signal, continuously optimize the SDF MLP network, and finally obtain a neural network model representing the scene surface. This network model estimates whether a spatial point is a scene surface point, and generates a smooth and detailed scene surface based on the scene surface points. Methods such as NC-SDF, ND-SDF, and DebSDF have good performance in terms of geometric details. In terms of rendering effect, volume rendering may result in blurred edges or unnatural transitions, resulting in a lack of rendering quality.

[0004] Gaussian-based scene reconstruction: That is, 2D image data is used to obtain an initial point cloud through Structure from Motion (SfM). The 3D Gaussian primitives are initialized with this point cloud, and the real 2D image data is used as the supervision signal to continuously train and optimize the attributes of the 3D Gaussian primitives, such as position, scale, color, etc., and finally obtain the optimal scene representation. Methods such as GigaGS and Dust-GS, in terms of rendering effects, due to the probability distribution property of the Gaussian, the scenes represented by 3D Gaussians are more likely to achieve rich details and soft lighting changes, making the scenes present more natural details and visual effects, which also determines that their rendering quality is better. In terms of geometric details, due to the volume size and ambiguity of the Gaussian, it is difficult to accurately fit complex geometric shapes, and it performs poorly in representing small local geometric details.

[0005] For the Signed Distance Field (SDF) scene reconstruction method, a GSDF scene reconstruction method has also been innovated, that is, by combining a bipartite architecture of 3D Gaussian and Signed Distance Field (SDF) to simultaneously improve the scene rendering quality and the accuracy of geometric reconstruction. The invention patent application with the application number 202410462822.5 discloses a Gaussian rendering and reconstruction method based on the prior guidance of the signed distance radiation field, which includes the following steps: Step 1: After performing ray tracing sampling on the input sequence of images, input them into the signed distance radiation field composed of two networks, namely the signed distance field and the color field, and use the signed distance radiation field to learn the SDF value and color value of the target object; Step 2: Integrate and render the image based on the volume rendering equation; Step 3: Calculate the L1 loss between the rendered image and the true value of the input image, and at the same time, regularly constrain the changes of the SDF value in the gradient direction and curl direction to continuously optimize the network parameters of the signed distance radiation field by forward propagating the gradient; Step 4: Initialize the three-dimensional Gaussian parameters and scene representation using the signed distance radiation field prior; Step 5: Perform Gaussian rendering and collision filtering on the three-dimensional Gaussian scene representation; Step 6: Calculate the necessity score based on the signed distance field and perform pruning to optimize the point cloud; Step 7: Calculate the color loss of Gaussian rendering, the regularization constraint of the central distribution of the Gaussian distribution, and the regularization constraint of the diagonal scaling matrix, and continuously optimize the three-dimensional Gaussian parameters by forward propagating the gradient; Step 8: Based on the optimized and converged signed distance radiation field and the three-dimensional Gaussian scene representation, generate high-quality geometric surface information and an image with high-fidelity and real-time Gaussian rendering results.

[0006] Similar to the above-mentioned prior art, no matter which three-dimensional scene expression method is adopted, it either has an advantage in reconstructing an accurate geometric model or is good at rendering high-quality images. As a result, in the process of three-dimensional scene reconstruction, there is often a contradiction between the rendering effect and the geometric reconstruction accuracy. If high-quality rendered images are generated, the details of the reconstructed geometric model will be insufficient. If the focus is on the reconstruction of a high-precision geometric model, the quality of the generated new perspective images will be poor. Therefore, it is necessary to solve the technical problem that geometric accuracy and rendering quality cannot be achieved simultaneously in the process of three-dimensional reconstruction, and provide a three-dimensional scene reconstruction method that has both accurate geometric details and can present a high-quality rendering effect. Summary of the Invention

[0007] The purpose of the present invention is to provide a three-dimensional scene and object reconstruction method, system, and device based on SDF and Gaussian fields in order to solve the technical problem that the three-dimensional scene reconstruction quality is not high due to the inability to achieve both geometric accuracy and rendering quality simultaneously in the process of three-dimensional reconstruction in the prior art.

[0008] The present invention specifically adopts the following technical solutions to achieve the above purpose: A three-dimensional scene and object reconstruction method based on SDF and Gaussian fields, comprising the following steps: Step 1, generate an initial three-dimensional model of the scene; First, use the first-frame real depth map , the first-frame real RGB image to initialize the TSDF voxel volume, and then use the real depth maps and real RGB images of subsequent frames to update the TSDF voxel volume to obtain the external camera parameters and the initial three-dimensional model of the scene; Step 2, train the SDF neural network; Use the initial three-dimensional model of the scene and the internal and external camera parameters in Step 1 to calculate the point cloud under each frame of the camera's perspective; interpolate between two adjacent interpolation sampling points on the light direction to obtain the scene surface points on the light ray ; Input the coordinates of each point in each frame of the point cloud , the coordinates of the scene surface points obtained by interpolation, and the light direction into the SDF neural network to train the SDF neural network; Step 3, extract the rough model of the scene grid; Perform grid division on the bounding box of the initial three-dimensional model of the scene obtained in Step 1, and use the coordinates of each grid point as the input of the trained SDF neural network to obtain the SDF prediction value and the rough rendering map of each grid point; Filter out the points with SDF predicted values close to 0 according to the SDF predicted values of the grid points, and use the Marching cubes algorithm to extract the rough model of the scene grid; Step 4, Gaussian rendering; Use the rendered rough image and the rough model of the scene grid obtained in Step 3 as the initial points of the Gaussian field, optimize the Gaussian basis elements, and obtain the final scene geometry model; Generate the final rendered image using the optimized Gaussian basis elements.

[0009] Furthermore, in Step 1, when initializing the TSDF voxel volume, the specific steps are as follows: Read the first frame of the real depth map and the first frame of the real RGB image , and use the camera internal parameters to back-project the image 2D points into the camera coordinate system to obtain the 3D point cloud data of the first frame : ; Detect the plane in the 3D point cloud data of the first frame , and use the point closest to the plane in the 3D point cloud of this frame as the origin of the world coordinate system. Take as the x-axis, take as the z-axis, and take the cross product of the x-axis and the z-axis to obtain the y-axis to get the world coordinate system; Convert the point cloud in the first frame of the camera coordinate system to the world coordinate system, and adjust the boundaries of the TSDF voxel volume according to the actual range of the point cloud to complete the initialization of the TSDF voxel volume.

[0010] Furthermore, in Step 1, when updating the TSDF voxel volume, the specific steps are as follows: Read the real depth map, real RGB map and camera internal parameters of the current frame and back-project the point cloud into the camera coordinate system. Use the global camera pose and camera internal parameters of the previous frame, render the fused scene through the ray casting algorithm to generate the depth rendering map and RGB rendering map of the fused scene, obtain the point cloud of the previous frame based on the depth rendering map and RGB rendering map, and then use the ICP algorithm to calculate the transformation matrix between the point cloud of the previous frame and the point cloud of the current frame , and finally use the transformation matrix between frames to multiply with the global camera pose of the previous frame to obtain the global camera pose of the current frame .

[0011] Furthermore, in Step 2, when performing interpolation, if the SDF value of the sampling point ​ and the sampling points of the SDF value If the product of the two is less than 0, interpolate these two sampling points to obtain the scene surface points on the ray .

[0012] Furthermore, in step 2, when training the SDF neural network, the total loss function is:[[]] ; ; ; ; ; where , ,[[ID=3,4]] all represent weight values; represents the RGB image loss, used for the consistency constraint between the real RGB image and the RGB rendering image, represents the number of images, represents the i-th frame of the real RGB image, represents the i-th frame of the RGB rendering image; represents the SDF gradient consistency loss, used for constraining the gradient consistency of the SDF, represents the gradient operator, represents the point cloud the sdf estimated value of the j-th point in represents the SDF prediction loss, used for constraining the SDF predicted value of the sampling points, represents the point cloud under each frame of the camera view, represents the point cloud the j-th point in represents the point cloud the sdf true value of the j-th point in represents the number of sampling points in the i-th frame of the image; represents the multi-view photometric consistency loss, used for supervising the extraction of the scene surface, represents the number of 11×11 image patches in the source grayscale image, represents the pixel in the reference grayscale image, represents the pixel in the source grayscale image, represents the 11×11 image patch centered on the pixel in the reference grayscale image, represents the 11×11 image patch centered on the pixel in the source grayscale image (with a different view from the reference image).

[0013] Furthermore, in Step 4, the specific steps of Gaussian rendering are as follows: Step 4-1: Interpolate the vertices of the triangular faces of the rough scene grid model obtained in Step 3 to obtain sampling points ; interpolate the vertex normals to obtain normal vectors ; Step 4-2: Offset the sampling points along the normal vector direction to obtain the center position of the Gaussian basis element ; Step 4-3: Use the square root of the distance between the center position and the nearest Gaussian basis element as the scaling value , and initialize the rotation , opacity and spherical harmonic function of this Gaussian basis element; Step 4-4: Optimize the Gaussian basis element using the RGB real image, the internal and external camera parameters to obtain the final scene geometry model; generate the final rendered image using the optimized Gaussian basis element.

[0014] Even further, when optimizing the Gaussian basis element, the total loss between the rendered image and the real image is: ; ; ; ; ; ; where, , , , , are constants; represents the RGB image loss, which is used for the consistency constraint between the real RGB image and the RGB rendered image; respectively represent the height, width and number of channels of the picture, represents the pixel value of the rendered picture, represents the pixel value of the picture taken by the camera; represents the mean square error loss, which is used to represent the mean of the squared errors of all pixel points; represents the perceptual similarity loss; represents the number of feature layers for calculating the loss, represents the rendered picture, represents the picture taken by the camera, Represents an image The depth of the characteristic layer number; Represents the Gaussian basis element scaling loss, used to prevent the Gaussian basis element from becoming elongated; Represents the maximum value of Gaussian basis element scaling, Represents the minimum value of Gaussian basis element scaling, Represents the specified maximum scaling value, Represents the maximum value of the ratio of the maximum to the minimum scaling value; Represents the normal consistency loss, used to align the normal with the gradient of the depth map to make the Gaussian achieve a smoother surface; The opacity weight of the th Gaussian basis element Represents the normal map derived from the rendered depth map, Represents the normal of the th Gaussian basis element within the camera frustum.

[0015] A 3D scene and object reconstruction system based on SDF and Gaussian fields, including: A scene 3D initial model generation module, which first initializes the TSDF voxel volume using the first-frame real depth map and the first-frame real RGB image , and then updates the TSDF voxel volume using the real depth maps and real RGB images of subsequent frames to obtain the external camera parameters and the scene 3D initial model; An SDF neural network training module, which uses the scene 3D initial model and the internal and external camera parameters in the scene 3D initial model generation module to calculate the point cloud for each camera view; interpolates between two adjacent interpolation sampling points on the light direction to obtain the scene surface points on the light ray ; Inputs the coordinates of each point in each frame of point cloud , the coordinates of the scene surface points obtained after interpolation and the light direction into the SDF neural network to train the SDF neural network; A scene grid rough model extraction module, which divides the bounding box of the scene 3D initial model obtained by the scene 3D initial model generation module into grids, takes the coordinates of each grid point as the input of the trained SDF neural network, and obtains the SDF prediction value and the rendered rough image of each grid point; Filters out the points with SDF prediction values close to 0 according to the SDF prediction values of the grid points, and uses the Marching cubes algorithm to extract the scene grid rough model; The Gaussian rendering module is used to use the rendering coarse image and scene mesh coarse model obtained by the scene mesh coarse model extraction module as the initialization point of the Gaussian field, optimize the Gaussian basis element, and obtain the final scene geometric model; The final rendered image is generated using the optimized Gaussian primitives.

[0016] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the steps of the above method.

[0018] The beneficial effects of the present invention are as follows: 1. In this invention, the real depth map and RGB image are used to update the voxel volume and obtain the camera extrinsic parameters and the initial three-dimensional model of the scene. Based on the SDF neural network and Gaussian sputtering technology, a refined scene mesh model and rendering image are generated, which effectively solves the technical problem of the inability to achieve both geometric accuracy and rendering quality in the three-dimensional reconstruction process, and significantly improves the reconstruction quality of the three-dimensional scene.

[0019] 2. In the present invention, RGB-D data is used to provide geometric constraints for the implicit model, so that the model has a better geometric prior, which speeds up the model training process and enables it to reach convergence in less time.

[0020] 3. In the present invention, the SDF neural network is used to effectively estimate the scene surface points, thereby constructing a refined scene geometry model.

[0021] 4. In the present invention, a multi-layer perception SDF field is constructed using the depth map and sparse point cloud data of each frame image. 3D Gaussian initialization is initialized along the surface of the geometric object to perform a multi-stage optimization strategy to solve the contradiction between the geometric accuracy and rendering quality of three-dimensional scene reconstruction, and ultimately achieve the generation of high-fidelity three-dimensional scenes with accurate geometric details.

[0022] 5. In the present invention, the probability distribution property of Gaussian is utilized to make it easier for the scene to achieve rich details and soft lighting changes, ultimately presenting more natural details and visual effects. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0024] Therefore, based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0025] Embodiment 1 This embodiment provides a three-dimensional scene and object reconstruction method based on SDF and Gaussian field, including the following steps: Step 1, generate an initial three-dimensional model of the scene; First, use the first frame of real depth map and the first frame of real RGB image to initialize the TSDF voxel volume, and then use the real depth maps and real RGB images of subsequent frames to update the TSDF voxel volume to obtain the external camera parameters and the initial three-dimensional model of the scene.

[0026] First, initialize the TSDF voxel volume. The specific steps are as follows: Read the first frame of real depth map and the first frame of real RGB image , and use the camera internal parameters to back-project the image 2D points into the coordinate system to obtain the 3D point cloud data of the first frame : ; Subsequently, detect the plane in the 3D point cloud data of the first frame , calculate the point closest to the plane in the 3D point cloud of this frame as the origin of the world coordinate system, and use as the x-axis, use as the z-axis, and the cross product of the x-axis and the z-axis to obtain the y-axis to obtain the world coordinate system.

[0027] With the world coordinate system, next, convert the point cloud in the first frame of camera coordinate system to the world coordinate system representation, and adjust the boundary of the TSDF voxel volume according to the actual range of the point cloud, thus completing the initialization of the TSDF voxel volume.

[0028] Secondly, update the TSDF voxel volume. The specific steps are as follows: Read the real depth map, real RGB map and camera internal parameters of the current frame and back-project the point cloud into the camera coordinate system. Use the global camera pose and camera internal parameters of the previous frame, and render the fused scene through the ray casting algorithm (this algorithm is an existing technology and does not require creative efforts) to generate the depth rendering map and RGB rendering map of the fused scene. Obtain the point cloud of the previous frame based on the depth rendering map and RGB rendering map, and then use the ICP algorithm (this algorithm is an existing technology and does not require creative efforts) to calculate the transformation matrix between the point cloud of the previous frame and the point cloud of the current frame , finally, the transformation matrix between frames is multiplied by the global camera pose of the previous frame to obtain the global camera pose of the current frame : .

[0029] Using the calculated global camera pose of the current frame, the point cloud of the current frame is fused into the TSDF voxel volume in the world coordinate system. During the fusion process, the SDF value of each voxel is calculated by subtracting the depth value of the voxel from the Euclidean distance from the starting point of the projected ray to the voxel. To improve the calculation efficiency and geometric accuracy, in this embodiment, only the SDF values of the voxels containing the scene surface are updated. The SDF values of the voxels outside the scene surface are initialized to 1, and the SDF values of the voxels inside the scene surface are initialized to -1, that is, the SDF is truncated to within the range. The SDF values of the voxels containing the scene surface in the fused TSDF voxel volume are approximately 0. Finally, the voxels with SDF values approximately 0 in the scene voxel volume are used to extract the mesh model of the scene (i.e., the initial 3D model of the scene) by the Marching Cubes algorithm.

[0030] The TSDF formula is expressed as: ; where represents the truncation distance; if , then this point is on the surface of the scene.

[0031] Step 2, training the SDF neural network and generating the scene geometric model; Using the initial 3D model of the scene and the internal and external camera parameters in Step 1, calculate the point cloud under each frame of the camera view; interpolate between two adjacent interpolation sampling points on the light direction to obtain the scene surface points on the light ray; Input the coordinates of each point in each frame of the point cloud , the coordinates of the scene surface points obtained by interpolation, and the light direction into the SDF neural network to train the SDF neural network.

[0032] Using the initial 3D model of the scene and the corresponding internal and external camera parameters in Step 1, calculate the point cloud under each frame of the camera view , and use the coordinates of each point in each frame of the point cloud as one of the inputs of the SDF neural network, and the SDF neural network will obtain these point clouds under each frame of the camera view Each point in the SDF value of Here, the premise assumption is that the point cloud obtained in step 1 is the real scene surface points. Under this assumption, the true SDF value of the scene surface points To make the SDF value of the points on the geometric surface be 0, use the loss to directly constrain the value of SDF to supervise the predicted value of the SDF network; to enhance the physical rationality of the scene geometric surface, use to constrain the gradient of SDF to ensure the SDF gradient consistency (i.e., satisfy the Eikonal equation).

[0033] Another input of the SDF neural network is the ray direction (that is, connecting each sampled pixel in each frame with the camera optical center O as the ray direction), the coordinate of a point on the ray For the convenience of description, it is turned to use to represent the sampled point. If the SDF value of the sampled point and the SDF value of the sampled point The product of the two is less than 0, then the scene surface point on the ray can be interpolated using these two acquired points Using these scene surface points to construct the geometric model of the scene; on this basis, use the multi-view photometric consistency loss to supervise the extracted scene surface to ensure that the scene in the visible part of the camera is geometrically consistent under adjacent multi-frame views.

[0034] That is, when training the SDF neural network, the total loss function is:[[]] ; Using the real RGB image as the supervision to obtain the loss The loss is expressed as:[[]] ; To enhance the physical rationality of the scene geometric representation, use to constrain the gradient of SDF to ensure the SDF gradient consistency. The loss is expressed as:[[]] ; To make the SDF value of the points on the geometric surface be 0, use to directly constrain the value of SDF to supervise the predicted value of the SDF neural network. The loss is expressed as:[[]] ; To ensure that the scenes in the visible part of the camera are geometrically consistent under adjacent multi-frame viewpoints, is used to supervise the extracted scene surface. The loss is expressed as: ; where , , all represent weight values; represents the RGB image loss, which is used for the consistency constraint between the real RGB image and the RGB rendered image, represents the number of images, represents the i-th frame real RGB image, represents the i-th frame RGB rendered image; represents the SDF gradient consistency loss, which is used to constrain the gradient consistency of the SDF, represents the gradient operator, represents the sdf estimated value of the j-th point in the point cloud ; represents the SDF prediction loss, which is used to constrain the SDF predicted value of the sampling points, represents the point cloud under each frame of camera viewpoint, represents the point cloud represents the j-th point in the point cloud, represents the point cloud represents the sdf ground truth value of the j-th point in the point cloud, represents the number of sampling points in the i-th frame image; represents the multi-view photometric consistency loss, which is used to supervise the extraction of the scene surface, represents the number of 11×11 image patches in the source grayscale image, represents the pixel in the reference grayscale image, represents the pixel in the source grayscale image, represents the 11×11 image patch centered on the pixel in the reference grayscale image, represents the 11×11 image patch centered on the pixel in the source grayscale image (with a different viewpoint from the reference image).

[0035] During interpolation, if the product of the SDF value of the sampling point and the SDF value of the sampling point is less than 0, then interpolate these two sampling points to obtain the scene surface point on the ray.

[0036] The SDF neural network adopts the existing neural network structure, with 39 neurons in the input layer, 8 neurons in the hidden layer, 256 neurons in each hidden layer, and 257 neurons in the output layer (one of which is the SDF value and the rest are feature vectors). The specific structure of the SDF neural network can be found in https: / / arxiv.org / pdf / 2205.15848 Implementation details in the experimental section.

[0037] Step 3, extracting the scene grid coarse model; The bounding box of the initial 3D model of the scene obtained in step 1 is meshed, and the coordinates of each grid point are used as the input of the trained SDF neural network to obtain the SDF prediction value of each grid point and render the coarse image.

[0038] After the coordinates of all grid points are input into the SDF neural network, the SDF prediction values of all grid points in the scene grid are obtained.

[0039] Based on the SDF prediction values of the grid points, points with SDF prediction values close to 0 are screened out (as to whether they are close to 0, those skilled in the art can determine it based on actual conditions without inventive effort), and a Marching cubes algorithm (that is, an existing algorithm can be used without inventive effort) is used to extract a coarse scene grid model.

[0040] Step 4, Gaussian rendering; Use the coarse rendering image and the coarse scene mesh model obtained in step 3 as the initialization points of the Gaussian field, optimize the Gaussian basis element, and obtain the final scene geometry model; The final rendered image is generated using the optimized Gaussian primitives.

[0041] The specific steps of Gaussian rendering are: Step 4-1, interpolate to obtain sampling points and normal vectors; Randomly sample 70% of the triangles in all the face elements of the scene in the scene mesh model, and select the three vertices on each triangle. The weights are randomly initialized as , use the initialized weights to interpolate the vertices and get a sampling point within the triangle face The coordinates of the vertex normal are also interpolated using this weight to obtain the normal vector ; Sampling point , normal vector Respectively expressed as: ; .

[0042] Step 4-2, randomly offset the sampling points; Randomly offset the sampling points along the normal vector direction by a random distance of to obtain the center position of the Gaussian basis element .

[0043] Step 4-3, initialize the Gaussian basis element; Use the square root of the distance between this point and the nearest Gaussian basis element as the scaling value , and randomly initialize the rotation , opacity and spherical harmonic function of this Gaussian basis element: ; ; wherein, represents the position of the Gaussian basis element, represents the position of the Gaussian basis element closest to the Gaussian basis element, represents the random offset distance.

[0044] Step 4-4, optimize the Gaussian basis element and render the image; Optimize the Gaussian basis element using the RGB real image and the internal and external camera parameters, and obtain the final scene geometry model; generate the final rendered image using the optimized Gaussian basis element.

[0045] The specific process is as follows: Step 4-4-1, optimize the Gaussian basis element; Input the i-th frame RGB real image and select the Gaussian basis elements within the corresponding frustum of the camera to save computational effort. Based on the internal camera parameters and the external camera parameters calculated in Step 1, project the Gaussian basis elements within this frustum onto the imaging plane of the camera.

[0046] Render the projected Gaussian basis elements based on the rasterization of image patches to obtain a rendered image, calculate the loss between the rendered image and the real image and perform backpropagation to optimize the position , rotation , opacity and spherical harmonic function attributes of the Gaussian basis element.

[0047] During the process of optimizing the Gaussian basis element, calculate the new position of the Gaussian basis element according to the weights of the three points on the triangular facet and the offset distance in Step 5-1 described above; calculate the rotation and area scaling of the corresponding triangular facets of the new scene network model and the initialized scene network model The changes are attached to the attributes of the Gaussian surface element under the current scene geometry. That is, the Gaussian basis element of the current scene geometry can be expressed as: ; ; ; Among them, represents the final position of the Gaussian basis element, represents the final scaling of the Gaussian basis element, represents the final rotation of the Gaussian basis element, represents the sampling point of the m-th triangular surface element and the normal vector.

[0048] Final position , final scaling , final rotation plus the opacity . The image under the current scene geometry can be rendered. Compare the currently rendered image with the RGB ground truth of the i-th frame to calculate the loss, the mean squared error loss, the perceptual similarity loss, the Gaussian basis element scaling loss, and the normal consistency loss .

[0049] When optimizing the Gaussian basis element, the total loss is expressed as: ; ; ; ; ; ; Among them, , , , , are constants; represents the RGB image loss, which is used for the consistency constraint between the real RGB image and the RGB rendered image; respectively represent the height, width, and number of channels of the image, represents the pixel value of the rendered image, represents the pixel value of the image captured by the camera; represents the mean squared error loss, which is used to represent the mean of the squared errors of all pixel points; Represents the perceptual similarity loss; Represents the number of feature layers for calculating the loss, Represents the rendered image, Represents the image captured by the camera, Represents the image of the depth of the feature layer; Represents the Gaussian basis element scaling loss, used to prevent the Gaussian basis element from becoming elongated; Represents the maximum value of the Gaussian basis element scaling, Represents the minimum value of the Gaussian basis element scaling, Represents the specified maximum scaling value, Represents the maximum value of the ratio of the maximum to the minimum scaling values; Represents the normal consistency loss, used to align the normal with the gradient of the depth map to make the Gaussian achieve a smoother surface; Represents the opacity weight of the th Gaussian basis element, Represents the normal map derived from the rendered depth map, Represents the normal of the

[0050] th Gaussian basis element within the camera frustum.

[0050] Continuously optimize the parameter weights , offset distance , scaling parameter , rotation parameter , spherical harmonic function and opacity through the above losses. After the training is completed, the final scene model can be obtained.

[0051] After the training and optimization of the Gaussian basis element are completed, the final scene geometry model can be obtained.

[0052] Step 4-4-2, generate the final rendered image; According to the internal and external camera parameters obtained during training, project the optimized center of the Gaussian basis element onto the camera imaging plane, calculate its two-dimensional position in the pixel space, where the shape feature of each high-speed basis element is determined by its final scaling parameter and rotation parameter jointly. Use the rasterization technique to perform weighted summation of the pixels for the Gaussian basis elements projected into the pixel space according to the distribution weight expected for the opacity of each basis element, use the optimized spherical harmonic function to model the color of the Gaussian basis element, and generate the final rendered image by mixing the opacity and color distribution of the Gaussian basis element.

[0053] Example 2 This embodiment provides a three-dimensional scene and object reconstruction system based on SDF and Gaussian fields, including: A scene three-dimensional initial model generation module, which is used to first initialize the TSDF voxel volume using the first-frame real depth map and the first-frame real RGB image , and then update the TSDF voxel volume using the real depth maps and real RGB images of subsequent frames to obtain the external camera parameters and the scene three-dimensional initial model.

[0054] First, initialize the TSDF voxel volume. The specific steps are as follows: Read the first-frame real depth map and the first-frame real RGB image , and use the camera internal parameters to back-project the image 2D points into the coordinate system to obtain the 3D point cloud data of the first frame : ; Subsequently, detect the plane in the 3D point cloud data of the first frame , calculate the point closest to the plane in the 3D point cloud of this frame as the origin of the world coordinate system, take as the x-axis, take as the z-axis, and obtain the y-axis by taking the cross product of the x-axis and the z-axis to obtain the world coordinate system.

[0055] With the world coordinate system, next, convert the point cloud in the first-frame camera coordinate system to the world coordinate system representation, and adjust the boundary of the TSDF voxel volume according to the actual range of the point cloud, thus completing the initialization of the TSDF voxel volume.

[0056] Secondly, update the TSDF voxel volume. The specific steps are as follows: Read the real depth map, real RGB map, and camera internal parameters of the current frame and back-project the point cloud into the camera coordinate system. Use the global camera pose and camera internal parameters of the previous frame, and render the fused scene through the ray casting algorithm (this algorithm is an existing technology and does not require creative labor) to generate the depth rendering map and RGB rendering map of the fused scene. Obtain the point cloud of the previous frame based on the depth rendering map and RGB rendering map, and then use the ICP algorithm (this algorithm is an existing technology and does not require creative labor) to calculate the transformation matrix between the point cloud of the previous frame and the point cloud of the current frame , and finally multiply the transformation matrix between frames and the global camera pose of the previous frame to obtain the global camera pose of the current frame: .

[0057] Using the calculated global camera pose of the current frame, fuse the point cloud of the current frame into the TSDF voxel volume in the world coordinate system. During the fusion process, the SDF value of each voxel is calculated by subtracting the depth value of the voxel from the Euclidean distance from the starting point of the projected ray to the voxel. To improve the calculation efficiency and geometric accuracy, in this embodiment, only the SDF values of the voxels containing the scene surface are updated, the SDF values of the voxels outside the scene surface are initialized to 1, and the SDF values of the voxels inside the scene surface are initialized to -1, that is, the SDF is truncated to the range of. In the fused TSDF voxel volume, the SDF values of the voxels containing the scene surface are approximately 0. Finally, the voxels with SDF values approximately 0 in the scene voxel volume are used to extract the mesh model of the scene (i.e., the initial 3D model of the scene) by the Marching Cubes algorithm.

[0058] The TSDF formula is expressed as: ; where represents the truncation distance; if , then this point is on the surface of the scene.

[0059] The SDF neural network training module is used to calculate the point cloud under the camera view of each frame by using the initial 3D model of the scene and the internal and external camera parameters in the initial 3D model generation module of the scene; interpolate between two adjacent interpolation sampling points on the light ray direction to obtain the scene surface point on the light ray; Input the coordinates of each point in each frame of point cloud , the coordinates of the scene surface point obtained by interpolation, and the light ray direction into the SDF neural network to train the SDF neural network.

[0060] Using the initial 3D model of the scene and the corresponding internal and external camera parameters in step 1, calculate the point cloud under the camera view of each frame. Take the coordinates of each point in each frame of point cloud as one of the inputs of the SDF neural network. The SDF neural network will obtain the SDF values of each point in these point clouds under the camera view of each frame. . The prerequisite here is that the point cloud obtained in step 1 is the real scene surface point. Under this assumption, the true SDF value of the scene surface point 。To make the SDF value of points on the geometric surface equal to 0, the loss is used to directly constrain the SDF value to supervise the predicted value of the SDF network; to enhance the physical rationality of the scene geometric surface, is used to constrain the gradient of the SDF to ensure the consistency of the SDF gradient (i.e., satisfy the Eikonal equation).

[0061] Another input of the SDF neural network is the ray direction (that is, the connection between the sampled pixel of each frame and the camera optical center O is used as the ray direction), and the coordinate of a point on the ray , for the convenience of description, is instead represented by to represent the sampled point. If the SDF value of the sampled point and the SDF value of the sampled point have a product less than 0, then the two sampled points can be used for interpolation to obtain the scene surface point on the ray . The scene geometric model is constructed using these scene surface points; on this basis, the multi-view photometric consistency loss is used to supervise the extracted scene surface to ensure that the scene in the visible part of the camera is geometrically consistent under adjacent multi-frame views.

[0062] That is, when training the SDF neural network, the total loss function is: ; Using the real RGB image as supervision to obtain the loss . The loss is expressed as: ; To enhance the physical rationality of the scene geometric representation, is used to constrain the gradient of the SDF to ensure the consistency of the SDF gradient. The loss is expressed as: ; To make the SDF value of points on the geometric surface equal to 0, is used to directly constrain the SDF value to supervise the predicted value of the SDF neural network during subsequent training. The loss is expressed as: ; To ensure that the scene in the visible part of the camera is geometrically consistent under adjacent multi-frame views, is used to supervise the extracted scene surface. The loss is expressed as: ; Among them, , , all represent weight values; represents the RGB image loss, which is used for the consistency constraint between the real RGB image and the RGB rendering image, represents the number of images, represents the i-th frame of the real RGB image, represents the i-th frame of the RGB rendering image; represents the SDF gradient consistency loss, which is used to constrain the gradient consistency of the SDF, represents the gradient operator, represents the point cloud the sdf estimated value of the j-th point in; represents the SDF prediction loss, which is used to constrain the SDF predicted value of the sampling points, represents the point cloud under the camera view of each frame, represents the point cloud the j-th point in, represents the point cloud the sdf true value of the j-th point in, represents the number of sampling points in the i-th frame image; represents the multi-view photometric consistency loss, which is used to supervise the extraction of the scene surface, represents the number of 11×11 image patches in the source grayscale image, represents the pixel in the reference grayscale image, represents the pixel in the source grayscale image, represents the 11×11 image patch centered on the pixel in the reference grayscale image, represents the 11×11 image patch centered on the pixel in the source grayscale image (with a different view from the reference image).

[0063] When performing interpolation, if the SDF value of the sampling point and the SDF value of the sampling point the product of the two is less than 0, then interpolate these two sampling points to obtain the scene surface point on the ray.

[0064] This SDF neural network adopts an existing neural network structure. Its input layer has 39 neurons, its hidden layer has 8 layers of neurons, each hidden layer has 256 neurons, and its output layer has 257 neurons (one of which is the SDF value, and the rest are feature vectors). For the specific structure of this SDF neural network, please refer to https: / / arxiv.org / pdf / 2205.15848Implementation details in the experimental section.

[0065] The scene grid coarse model extraction module is used to grid the bounding box of the scene 3D initial model obtained by the scene 3D initial model generation module, use the coordinates of each grid point as the input of the trained SDF neural network, obtain the SDF prediction value of each grid point, and render the coarse image.

[0066] After the coordinates of all grid points are input into the SDF neural network, the SDF prediction values of all grid points in the scene grid are obtained.

[0067] Based on the SDF prediction values of the grid points, points with SDF prediction values close to 0 are screened out (as to whether they are close to 0, those skilled in the art can determine it based on actual conditions without inventive effort), and a Marching cubes algorithm (that is, an existing algorithm can be used without inventive effort) is used to extract a coarse scene grid model.

[0068] The Gaussian rendering module is used to use the rendering coarse image and scene mesh coarse model obtained by the scene mesh coarse model extraction module as the initialization point of the Gaussian field, optimize the Gaussian basis element, and obtain the final scene geometric model.

[0069] The specific steps of Gaussian rendering are: Step 4-1, interpolate to obtain sampling points and normal vectors; Randomly sample 70% of the triangles in all the face elements of the scene in the scene mesh model, and select the three vertices on each triangle. The weights are randomly initialized as , use the initialized weights to interpolate the vertices and get a sampling point within the triangle face The coordinates of the vertex normal are also interpolated using this weight to obtain the normal vector ; Sampling point , normal vector Respectively expressed as: ; .

[0070] Step 4-2, random offset of sampling points; The sampling point Along the normal vector The sampling points are randomly offset in the direction, and the offset distance is , and get the center position of the Gaussian element .

[0071] Step 4-3, initialize the Gaussian primitives; Use the square root of the distance between this point and the nearest Gaussian basis element as the scaling value and randomly initialize the rotation of this Gaussian basis element , opacity and spherical harmonics : ; ; wherein represents the position of the Gaussian basis element, represents the position of the Gaussian basis element closest to this Gaussian basis element, represents the random offset distance

[0072] Step 4-4, optimize the Gaussian basis element and render the image; Optimize the Gaussian basis element using the RGB real image and the internal and external camera parameters, and obtain the final scene geometry model; generate the final rendered image using the optimized Gaussian basis element

[0073] The specific process is as follows: Step 4-4-1, optimize the Gaussian basis element; Input the i-th frame RGB real image and select the Gaussian basis elements within the corresponding frustum in the camera to save computational effort. Based on the internal camera parameters and the external camera parameters calculated in Step 1, project the Gaussian basis elements within this frustum onto the imaging plane of the camera

[0074] Render the projected Gaussian basis elements based on the rasterization of image patches to obtain a rendered image, calculate the loss between the rendered image and the real image and perform backpropagation to optimize the parameters of the position , rotation , opacity and spherical harmonics attributes of the Gaussian basis element

[0075] During the process of optimizing the Gaussian basis element, calculate the new position of the Gaussian basis element according to the weights of the three points on the triangular facet in the aforementioned Step 5-1 and the offset distance ; calculate the changes in the rotation and the scaling of the area of the corresponding triangular facets of the new scene network model and the initialized scene network model, and attach them to the attributes of the Gaussian facets under the current scene geometry, that is, the Gaussian basis element under the current scene geometry can be expressed as: ; ; ; wherein represents the final position of the Gaussian primitive, represents the final scaling of the Gaussian basis, represents the final rotation of the Gaussian basis, Indicates the mth triangle surface sampling point The normal vector of .

[0076] Final position , Final Scaling , final rotation Add opacity , you can render the image under the current scene geometry. Compare the currently rendered image with the RBG real image of the i-th frame loss, Mean square error loss, Perceptual similarity loss, Gaussian base scaling loss and normal consistency loss .

[0077] When optimizing the Gaussian primitive, the total loss Expressed as: ; ; ; ; ; ; in, 、 、 、 、 is a constant; Represents RGB image loss, which is used to constrain the consistency between the real RGB image and the RGB rendering image; Represents the height, width and number of channels of the image respectively, Represents the pixel value of the rendered image, Indicates the pixel value of the picture taken by the camera; Represents the mean square error loss, which is used to represent the mean of the square errors of all pixels; represents the perceptual similarity loss; Indicates the number of feature layers for calculating loss, Represents a rendered image. Indicates that the camera takes a picture. Representing an image No. The depth of the feature layer; represents the Gaussian primitive scaling loss, which is used to prevent the Gaussian primitive from becoming elongated; Represents the maximum value of the Gaussian basis element scaling, represents the minimum value of the Gaussian basis element scaling, represents the specified maximum scaling value, represents the maximum value of the ratio of the maximum to the minimum scaling values; represents the normal consistency loss, which is used to align the normal with the gradient of the depth map to make the Gaussian achieve a smoother surface; represents the opacity weight of the -th Gaussian basis element and represents the normal map derived from the rendered depth map, represents the normal of the -th Gaussian basis element within the camera frustum.

[0078] The parameter weights are continuously optimized through the above losses , offset distance , scaling parameters , rotation parameters , spherical harmonics and opacity . After the training is completed, the final scene model can be obtained.

[0079] After the training and optimization of the Gaussian basis elements are completed, the final scene geometry model can be obtained.

[0080] Step 4-4-2, generate the final rendered image; According to the internal and external camera parameters obtained during training, project the optimized Gaussian basis element centers onto the camera imaging plane, calculate their two-dimensional positions in the pixel space, where the shape features of each high-speed basis element are determined by its final scaling parameter and rotation parameter jointly. Use rasterization technology to perform weighted summation of the pixels for the Gaussian basis elements projected into the pixel space according to the distribution weights expected for the opacity of each basis element, use the optimized spherical harmonics to perform color modeling on the Gaussian basis elements, and generate the final rendered image by mixing the opacity and color distributions of the Gaussian basis elements.

[0081] Embodiment 3 A computer device includes a memory and a processor. When the computer program stored in the memory is executed by the processor, the processor executes the steps of the three-dimensional scene and object reconstruction method based on SDF and Gaussian fields.

[0082] Among them, the computer device can be a computing device such as a desktop computer, a notebook, a palm computer, or a cloud server. The computer device can interact with the user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device, etc.

[0083] The memory at least includes one type of readable storage medium, and the readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or D-interface display memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Of course, the memory can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the memory is commonly used to store the operating system installed on the computer device and various application software, such as the program code of the three-dimensional scene and object reconstruction method based on SDF and Gaussian fields. In addition, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0084] In some embodiments, the processor can be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the three-dimensional scene and object reconstruction method based on SDF and Gaussian fields.

[0085] Embodiment 4 A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor is caused to execute the steps of the three-dimensional scene and object reconstruction method based on SDF and Gaussian fields.

[0086] Among them, the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to cause the at least one processor to execute the steps of the three-dimensional scene and object reconstruction method based on SDF and Gaussian fields as described above.

[0087] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the three-dimensional scene and object reconstruction method based on SDF and Gaussian fields described in the embodiments of the present application.

Claims

1. A three-dimensional scene and object reconstruction method based on SDF and Gaussian fields, characterized in that Including the following steps: Step 1, generating an initial 3D model of the scene; First, use the first-frame real depth map and the first-frame real RGB image to initialize the TSDF voxel volume, and then use the real depth maps and real RGB images of subsequent frames to update the TSDF voxel volume to obtain the external camera parameters and the initial 3D scene model; Step 2, training the SDF neural network; Using the 3D initial model of the scene in Step 1 and the internal and external camera parameters, calculate the point cloud under the camera view for each frame; for the light direction Interpolate between two adjacent interpolation sampling points on the light ray to obtain the scene surface points on the light ray ; For each frame of point cloud each point coordinates, the coordinates of the scene surface points obtained after interpolation coordinates and light direction are input into the SDF neural network for training the SDF neural network; Step 3, extracting a coarse scene mesh model; Performing mesh division on the bounding box of the initial 3D model of the scene obtained in Step 1, using the coordinates of each grid point as the input of the trained SDF neural network, obtaining the SDF prediction value of each grid point and rendering a coarse image; Selecting the points with SDF prediction values close to 0 according to the SDF prediction values of the grid points, and using the Marching cubes algorithm to extract the coarse scene mesh model; Step 4, Gaussian rendering; Using the rendered coarse image and the coarse scene mesh model obtained in Step 3 as the initialization points of the Gaussian field, optimizing the Gaussian basis elements to obtain the final scene geometry model; Using the optimized Gaussian basis elements to generate the final rendered image.

2. The three-dimensional scene and object reconstruction method based on SDF and Gaussian field according to claim 1, characterized in that In Step 1, when initializing the TSDF voxel volume, the specific steps are as follows: Read the first frame of real depth map , the first frame of real RGB image , and use the camera internal parameters to back-project the image 2D points into the camera coordinate system to obtain the 3D point cloud data of the first frame : ； Detect the 3D point cloud data of the first frame in the plane , and use the point closest to the plane in the 3D point cloud of this frame as the origin of the world coordinate system. Use as the x-axis, and use as the z-axis. The cross product of the x-axis and the z-axis gives the y-axis, thus obtaining the world coordinate system; Converting the point cloud in the first frame camera coordinate system to the world coordinate system, and adjusting the boundary of the TSDF voxel volume according to the actual range of the point cloud to complete the initialization of the TSDF voxel volume.

3. A three-dimensional scene and object reconstruction method based on SDF and Gaussian fields according to claim 1, characterized in that In Step 1, when updating the TSDF voxel volume, the specific steps are as follows: Read the real depth map, real RGB map of the current frame and the camera intrinsics, and back-project them into the point cloud in the camera coordinate system. Use the global camera pose and camera intrinsics of the previous frame, and render the fused scene through the ray casting algorithm to generate the depth rendering map and RGB rendering map of the fused scene. Obtain the point cloud of the previous frame based on the depth rendering map and RGB rendering map, and then use the ICP algorithm to calculate the point cloud of the previous frame and the point cloud of the current frame to obtain the transformation matrix . Finally, use the transformation matrix between frames and the global camera pose of the previous frame to multiply and obtain the global camera pose of the current frame .

4. The three-dimensional scene and object reconstruction method based on SDF and Gaussian field according to claim 1, characterized in that In step 2, when performing interpolation, if the product of the SDF value of the sampling point and the SDF value of the sampling point is less than 0, then interpolate between these two sampling points to obtain the scene surface point on the ray and the SDF value of the sampling point . .

5. A three-dimensional scene and object reconstruction method based on SDF and Gaussian fields according to claim 4, characterized in that In step 2, when training the SDF neural network, the total loss function is as follows: ； ； ； ； ； Among them, , , all represent weight values; represents the RGB image loss, which is used for the consistency constraint between the real RGB image and the RGB rendering image, represents the number of images, represents the i-th frame of the real RGB image, represents the i-th frame of the RGB rendering image; represents the SDF gradient consistency loss, which is used to constrain the gradient consistency of the SDF, represents the gradient operator, represents the point cloud the sdf estimated value of the j-th point in represents the SDF prediction loss, which is used to constrain the SDF predicted value of the sampling points, represents the point cloud under the camera view of each frame, represents the point cloud the j-th point in represents the point cloud the sdf true value of the j-th point in represents the number of sampling points in the i-th frame of the image; represents the multi-view photometric consistency loss, which is used to supervise the extraction of the scene surface, represents the number of 11×11 image patches in the source grayscale image, represents the pixel in the reference grayscale image, represents the pixel in the source grayscale image, represents the 11×11 image patch centered on the pixel in the reference grayscale image, represents the 11×11 image patch centered on the pixel in the source grayscale image (with a different view from the reference image).

6. A three-dimensional scene and object reconstruction method based on SDF and Gaussian field according to claim 1, characterized in that In Step 4, the specific steps of Gaussian rendering are as follows: Step 4-1: Interpolate the vertices of the triangular faces of the rough scene grid model obtained in Step 3 to obtain sampling points ; interpolate the vertex normals to obtain normal vectors ; Step 4-2, shift the sampling point along the normal vector direction to obtain the center position of the Gaussian basis element ; Step 4-3, according to the central position Take the square root of the distance between the nearest Gaussian basis element as the scaling value , and initialize the rotation of this Gaussian basis element , opacity and spherical harmonics ; Step 4-4, using the RGB real image and the internal and external camera parameters to optimize the Gaussian basis elements to obtain the final scene geometry model; using the optimized Gaussian basis elements to generate the final rendered image.

7. A three-dimensional scene and object reconstruction method based on SDF and Gaussian fields according to claim 6, characterized in that, When optimizing the Gaussian basis elements, the total loss between the rendered image and the real image is: ； ； ； ； ； ； in, 、 、 、 、 is a constant; Represents RGB image loss, which is used to constrain the consistency between the real RGB image and the RGB rendering image; Represents the height, width and number of channels of the image respectively, Represents the pixel value of the rendered image, Indicates the pixel value of the picture taken by the camera; Represents the mean square error loss, which is used to represent the mean of the square errors of all pixels; represents the perceptual similarity loss; Indicates the number of feature layers for calculating loss, Represents a rendered image. Indicates that the camera takes a picture. Representing an image No. The depth of the feature layer; represents the Gaussian primitive scaling loss, which is used to prevent the Gaussian primitive from becoming elongated; represents the maximum value of the Gaussian basis element scaling, represents the minimum value of Gaussian basis scaling, Indicates the maximum scaling value specified, The maximum value representing the ratio of the maximum to the minimum zoom value; Represents the normal consistency loss, which is used to align the normal with the gradient of the depth map to make the Gaussian achieve a smoother surface; Indicates the Opacity weight of the Gaussian primitives , represents the normal map derived from the rendered depth map, Indicates the first The normal of the Gaussian primitive.

8. A three-dimensional scene and object reconstruction system based on SDF and Gaussian fields, characterized in that, Including: A scene three-dimensional initial model generation module, which is used to first initialize the TSDF voxel volume using the first-frame real depth map , the first-frame real RGB image to update the TSDF voxel volume using the real depth maps and real RGB images of subsequent frames, and obtain the external camera parameters and the initial three-dimensional model of the scene; The SDF neural network training module is used to generate the 3D initial model of the scene and the internal and external camera parameters in the 3D initial model generation module of the scene, calculate the point cloud under the perspective of each frame of the camera; for the light direction Interpolate between two adjacent interpolation sampling points on the light ray to obtain the scene surface points on the light ray ; For each frame of point cloud each point coordinates, the coordinates of the scene surface points obtained after interpolation coordinates and light direction are input into the SDF neural network for training the SDF neural network; A coarse scene mesh model extraction module, which is used to perform mesh division on the bounding box of the initial 3D model of the scene obtained by the initial 3D model generation module of the scene, using the coordinates of each grid point as the input of the trained SDF neural network, obtaining the SDF prediction value of each grid point and rendering a coarse image; Selecting the points with SDF prediction values close to 0 according to the SDF prediction values of the grid points, and using the Marching cubes algorithm to extract the coarse scene mesh model; A Gaussian rendering module, which is used to use the rendered coarse image and the coarse scene mesh model obtained by the coarse scene mesh model extraction module as the initialization points of the Gaussian field, optimize the Gaussian basis elements to obtain the final scene geometry model; Using the optimized Gaussian basis elements to generate the final rendered image.

9. A computer device, characterized in that: Including a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Storing a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

Images