High-precision three-dimensional scene and object reconstruction method, system, equipment and medium

Abstract

The invention discloses a high-precision three-dimensional scene and object reconstruction method, system, equipment and medium, belongs to three-dimensional scene reconstruction in the technical field of computer vision, and aims to solve the technical problem of low three-dimensional scene reconstruction quality in the prior art. The method comprises the steps of calculating external parameters of a camera, determining coordinates and colors of sampling points closest to the surface of a scene, updating and optimizing parameters and generating a scene geometric model and a rendering graph, the rendering graph is generated based on an SDF MLP network and a Color MLP network in combination with a Gaussian sputtering technology, and parameter optimization is carried out on Gaussian primitives, the SDF MLP network and the Color MLP network by utilizing the generated rendering graph and a real graph. And the Gaussian primitive after parameter optimization, the SDF MLP network and the Color MLP network are utilized to finally generate a high-precision scene geometric model and a new visual angle image with high fidelity rendering quality, so that the technical problem that geometric precision and rendering quality cannot be obtained at the same time in a three-dimensional reconstruction process is effectively solved, and the reconstruction quality of a three-dimensional scene is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, relates to the reconstruction of a three-dimensional scene, and particularly relates to a method, system, device and medium for high-precision three-dimensional scene and object reconstruction. Background Technique

[0002] In recent years, three-dimensional scene reconstruction technology has achieved rapid development in the fields of computer vision and graphics, and has shown broad application potential in fields such as virtual reality, augmented reality, film production, game development, and architectural design. The most basic purpose of scene reconstruction is to recover the geometric structure of a complete three-dimensional scene from two-dimensional images and other sensor data. Among them, the precise geometric structure of the scene and the rendering quality of new viewpoints are the keys to enhancing the user experience in the above application fields.

[0003] In order to achieve a more realistic three-dimensional scene rendering effect, it is necessary to reconstruct a three-dimensional scene model with a high-precision geometric structure. However, scene reconstruction does face some significant challenges in terms of precise geometric structure and new viewpoint rendering. The existing methods for scene reconstruction mainly include the following aspects: Scene reconstruction based on SDF (Signed Distance Field): That is, starting from the pixels of a two-dimensional image, emit rays and sample points, input the coordinates of the sample points into the SDF MLP network, use this network to predict the SDF value of the sample points, use the Eikonal equation as the constraint of SDF and RGB as the color supervision signal, continuously optimize the SDF MLP network, and finally obtain a neural network model representing the scene surface. This network model estimates whether a spatial point is a scene surface point, and generates a smooth and detailed scene surface based on the scene surface points. Methods such as NC-SDF, ND-SDF, and DebSDF have good performance in terms of geometric details, but need to be optimized for each scene and lack the generalization ability for new scenes. In terms of rendering effects, volume rendering may result in blurred edges or unnatural transitions, resulting in a lack of rendering quality.

[0004] Gaussian-based scene reconstruction: that is, using 2D image data to recover structure from motion (SfM) to obtain an initialized point cloud, using this point cloud to initialize 3D Gaussian primitives, using real 2D image data as a supervisory signal, and continuously training and optimizing the position, scale, color and other properties of the 3D Gaussian primitives to ultimately obtain the optimal scene representation. In terms of rendering effects, methods such as GigaGS and Dust-GS are more likely to achieve rich details and soft lighting changes in scenes represented by 3D Gaussians due to the probability distribution properties of Gaussians, making it easier for scenes to present more natural details and visual effects, which also determines their superior rendering quality. In terms of geometric details, due to the size and fuzziness of Gaussians, it is difficult to accurately fit complex geometric shapes, and their performance is insufficient when representing small local geometric details.

[0005] For the SDF scene reconstruction method, a GSDF scene reconstruction method has also been innovated, that is, by combining the dual-branch architecture of 3D Gaussian and signed distance field SDF, to simultaneously improve the scene rendering quality and the accuracy of geometric reconstruction. The invention patent application with application number 202410462822.5 discloses a Gaussian rendering and reconstruction method based on the prior guidance of signed distance radiation field, which includes the following steps: Step 1: After ray tracing sampling of the input sequence image, the signed distance radiation field composed of two networks, signed distance field and color field, is input, and the SDF value and color value of the target object are learned using the signed distance radiation field; Step 2: Render the image based on the integral of the volume rendering equation; Step 3: Calculate the L1 loss of the rendered image and the true value of the input image, and at the same time regularly constrain the change of the SDF value in the gradient direction and the curl direction, so as to pass the gradient forward and continuously optimize the symbol Step 4: Initialize the three-dimensional Gaussian parameters and scene representation using the signed distance radiation field prior; Step 5: Perform Gaussian rendering and collision filtering on the three-dimensional Gaussian scene representation; Step 6: Calculate the necessity score based on the signed distance field and perform pruning to optimize the point cloud; Step 7: Calculate the color loss of Gaussian rendering, the center distribution regularization constraint of the Gaussian distribution and the regularization constraint of the diagonal scaling matrix, and pass the gradient forward to continuously optimize the three-dimensional Gaussian parameters; Step 8: Based on the signed distance radiation field and the three-dimensional Gaussian scene representation after optimization convergence, generate images with high-quality geometric surface information and high-fidelity, real-time Gaussian rendering results.

[0006] Similar to the above-mentioned prior art, no matter which three-dimensional scene expression method is adopted, it either has an advantage in reconstructing an accurate geometric model or is good at rendering high-quality images. As a result, in the process of three-dimensional scene reconstruction, there is often a contradiction between the rendering effect and the geometric reconstruction accuracy. If high-quality rendered images are generated, the details of the reconstructed geometric model will be insufficient. If the focus is on the reconstruction of a high-precision geometric model, the quality of the generated new perspective images will be poor. Therefore, it is necessary to solve the technical problem that geometric accuracy and rendering quality cannot be achieved simultaneously in the process of three-dimensional reconstruction, and provide a three-dimensional scene reconstruction method that has both accurate geometric details and can present a high-quality rendering effect. Summary of the Invention

[0007] The purpose of the present invention is to provide a high-precision three-dimensional scene and object reconstruction method, system, device and medium in order to solve the technical problem that the geometric accuracy and rendering quality cannot be achieved simultaneously in the prior art during the three-dimensional reconstruction process, resulting in low quality of three-dimensional scene reconstruction.

[0008] The present invention specifically adopts the following technical solutions to achieve the above purpose: A high-precision three-dimensional scene and object reconstruction method includes the following steps: Step 1, calculate the extrinsic parameters of the camera; Calculate the extrinsic parameters of the camera according to the real images, and the real images include real depth maps and real RGB maps ; Step 2, determine the coordinates and colors of the sampling points closest to the scene surface; Perform ray casting on the pixels of each frame of real RGB map, and input the coordinates of the sampling points closest to the scene surface within the upsampling range of the ray into the SDF MLP network. The SDF MLP network outputs the SDF prediction value and the feature vector of the sampling point closest to the scene surface; Input the direction of the ray , the coordinates of the sampling point closest to the scene surface , the gradient of the SDF prediction value , and the feature vector into the Color MLP network. The Color MLP network predicts the radiance of the sampling point closest to the scene surface and obtains the color of the sampling point closest to the scene surface; Step 3, parameter update and optimization; Use the coordinates and colors of the sampling points closest to the scene surface to initialize the position , shape , opacity λ, spherical harmonic function , select the Gaussian basis elements within the corresponding camera frustum, and project the Gaussian basis elements within the frustum onto the imaging plane of the camera based on the camera internal parameters and the camera external parameters calculated in step 1; For the Gaussian basis elements projected into the pixel space, perform weighted summation on the pixels according to the opacity λ and distribution weight of each Gaussian basis element; use spherical harmonic functions Perform color modeling on the Gaussian basis elements, and generate the final rendered image by mixing the opacity and color distribution of the Gaussian basis elements. The rendered image includes RGB rendered images, depth rendered images, and normal rendered images; Utilize the rendered image and the real image to update and optimize the position of the Gaussian basis elements, shape, opacity λ, spherical harmonic functions attributes, as well as the parameters of the SDF MLP network and the Color MLP network; Step 4, generate the scene geometry model and rendered images; Input the grid points layer by layer into the optimized SDF MLP network. The SDF MLP network outputs the SDF prediction values of the grid points, and filter out the points whose SDF prediction values are close to 0 according to the SDF prediction values of the grid points; Use the Marching cubes algorithm to extract the grid surface from the SDF prediction values of the grid points to obtain the scene geometry model; Input the camera internal parameters, external parameters of the corresponding view, and the attributes of the optimized Gaussian basis elements, retain all the Gaussian basis elements within the camera frustum of the corresponding view, and project the Gaussian basis elements within the frustum onto the imaging plane of the camera; For the Gaussian basis elements projected into the pixel space, perform weighted summation on the pixels according to the optimized opacity λ and distribution weight of each Gaussian basis element; Use the optimized spherical harmonic functions

[0009] Further, when calculating the camera external parameters in step 1, the specific method is: Step 1-1, read the first-frame real image, and use the camera internal parameters to back-project the image into the camera coordinate system to obtain the 3D point cloud data of the first frame : ; Among them, represents the first-frame real depth map, represents the first-frame real RGB map; Step 1-2, detect the plane in the first-frame 3D point cloud , calculate the point closest to the plane in the point cloud of this frame as the origin of the world coordinate system, and As the x-axis, As the z-axis, the x-axis and the z-axis are cross-producted to get the y-axis, and the world coordinate system is obtained; Steps 1-3: Convert the point cloud of the first frame of real image in the camera coordinate system to the world coordinate system, and adjust the boundary of the TSDF voxel volume according to the actual range of the point cloud to complete the initialization of the TSDF voxel volume; Steps 1-4 use the real image of subsequent frames to update the TSDF voxel volume and obtain the camera extrinsic parameters.

[0010] Furthermore, in steps 1-4, the specific steps for updating the TSDF voxel volume are: Read the real depth map, real RGB map and camera intrinsic parameters of the current frame and back-project them into the point cloud in the camera coordinate system. Use the global camera pose and camera intrinsic parameters of the previous frame to render the fused scene through the ray casting algorithm to generate the depth rendering and RGB rendering of the fused scene. Based on the depth rendering and RGB rendering, get the point cloud of the previous frame, and then use the ICP algorithm to calculate the point cloud of the previous frame. With the current frame point cloud The transformation matrix between , and finally use the transformation matrix between frames Global camera pose with the previous frame Multiply to get the global camera pose of the current frame .

[0011] Furthermore, in step 2, the specific method for determining the scene surface sampling points is: Perform ray casting on the pixels of each frame of the real RGB image and connect the camera optical center and pixels in the frame cast light outward in the direction ; Find two sampling points on the ray that meet the following filtering conditions 、 , and these sampling points form a set , find the set There are points with zero SDF between the sampling points, which are the sampling points on the scene surface; The filter criteria are: ; in, Represents the estimated SDF value of the sampling point on the ray passing through the k-th pixel in the i-th frame image, Represents the estimated SDF of another sampling point on the ray passing through the k-th pixel in the i-th frame image.

[0012] Going further, looking for sets When there is a point where the SDF is zero between the sampling points, the specific method is: Interpolate between adjacent sampling points in the set to obtain the set : ; ; Find the interpolation points between all sampling points that meet the screening conditions in the set and extract the interpolation point closest to the camera from the set which is the sampling point on the scene surface: ; wherein represents the interpolation point on the two sampling points on the light ray passing through the k-th pixel in the i-th frame of image, represents the sampling point on the light ray passing through the k-th pixel in the i-th frame of image, represents the other sampling point on the light ray passing through the k-th pixel in the i-th frame of image, represents the SDF estimated value of the sampling point on the light ray passing through the k-th pixel in the i-th frame of image, represents the SDF estimated value of the other sampling point on the light ray passing through the k-th pixel in the i-th frame of image.

[0013] Furthermore, when optimizing the parameters in step 3, the total loss is: [[ID=(42]]; ; ; ; ; ; wherein all represent weight values; represents the SDF prediction loss, which is used to constrain the SDF predicted value of the sampling points output by the SDFMLP network, represents the coordinates of the sampling point on the light ray passing through the k-th pixel in the i-th frame of image, represents the sampling point set, represents the number of sampling points on the light ray passing through the k-th pixel in the i-th frame of image, represents the SDF estimated value of the sampling point on the light ray passing through the k-th pixel in the i-th frame of image; represents the SDF gradient consistency loss, which is used to constrain the gradient consistency of the SDF, represents the gradient operator; Denotes the RGB image loss, which is used for the consistency constraint between the real RGB image and the RGB rendered image. Denotes the number of images. Denotes the i-th frame of the real RGB image. Denotes the i-th frame of the RGB rendered image. Denotes the depth image loss, which is used for the consistency constraint between the real depth image and the depth rendered image. Denotes the i-th frame of the real depth image. Denotes the i-th frame of the depth rendered image. Denotes the normal image loss, which is used to constrain the normal consistency between the normal map calculated from the depth map by computing the gradient and the normal rendered image. Denotes the gradient of the i-th frame of the real depth image. Denotes the i-th frame of the normal rendered image.

[0014] A high-precision three-dimensional scene and object reconstruction system, comprising: An external camera parameter calculation module, which is used to calculate the external camera parameters according to the real images, and the real images include the real depth image and the real RGB image ; A module for determining the coordinates and colors of the sampling points closest to the scene surface, which is used to perform ray casting on the pixels of each frame of the real RGB image, and input the coordinates of the sampling points closest to the scene surface within the upsampling range of the ray into the SDF MLP network. The SDF MLP network outputs the SDF prediction value and the feature vector of the sampling points closest to the scene surface; Input the direction of the ray , the coordinates of the sampling points closest to the scene surface , the gradient of the SDF prediction value , and the feature vector into the Color MLP network. The Color MLP network predicts the radiance of the sampling points closest to the scene surface and obtains the colors of the sampling points closest to the scene surface; A parameter update and optimization module, which is used to initialize the positions , shapes , opacities λ, and spherical harmonics of the Gaussian basis elements by using the coordinates and colors of the sampling points closest to the scene surface, select the Gaussian basis elements within the corresponding camera frustum, and project the Gaussian basis elements within the frustum onto the imaging plane of the camera based on the internal camera parameters and the external camera parameters calculated by the external camera parameter calculation module; For the Gaussian basis elements projected into the pixel space, perform weighted summation on the pixels according to the opacity λ and the distribution weights of each Gaussian basis element; Use spherical harmonics Perform color modeling on the Gaussian primitives and generate the final rendered image by mixing the opacity and color distribution of the Gaussian primitives. The rendered image includes RGB rendering, depth rendering, and normal rendering. Use rendered images and real images to determine the position of Gaussian primitives ,shape , opacity λ, spherical harmonics Update and optimize the attributes and parameters of the SDF MLP network and the Color MLP network; The scene geometry model and rendering generation module is used to input grid points layer by layer into the optimized SDF MLP network. The SDF MLP network outputs the SDF prediction values of the grid points and filters out points with SDF prediction values close to 0 based on the SDF prediction values of the grid points. The Marching cubes algorithm is used to extract the grid surface from the SDF prediction values of the grid points to obtain the scene geometry model. Input the camera intrinsic parameters, extrinsic parameters and properties of the optimized Gaussian primitives of the corresponding viewing angle, retain all Gaussian primitives within the camera cone of the corresponding viewing angle, and project the Gaussian primitives within the cone to the imaging plane of the camera; for the Gaussian primitives projected to the pixel space, perform weighted summation on the pixels according to the optimized opacity λ and distribution weight of each Gaussian primitive; use the optimized spherical harmonic function Color modeling is performed on the Gaussian primitives, and by mixing the opacity and color distribution of the Gaussian primitives, a high-fidelity new perspective rendering is obtained.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0016] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the steps of the above method.

[0017] The beneficial effects of the present invention are as follows: 1. In the present invention, a rendering image is generated based on the SDF MLP network and the Color MLP network in combination with Gaussian sputtering technology, and the parameters of the Gaussian primitives, SDF MLP network, and Color MLP network are optimized using the generated rendering image and the real image. The parameter-optimized Gaussian primitives, SDF MLP network, and Color MLP network are finally used to generate a high-precision scene geometry model and a new perspective image with high-fidelity rendering quality, effectively solving the technical problem of the inability to achieve both geometric accuracy and rendering quality in the 3D reconstruction process, and significantly improving the reconstruction quality of the 3D scene.

[0018] 2. In the present invention, the depth data of each frame of pixels is used to provide a relatively accurate sampling range for the sampling points, avoiding the introduction of complex sampling strategies, and thus accelerating the model training process.

[0019] 3. In the present invention, by utilizing the physical properties of SDF and cooperating with the interpolation strategy, the sampling points are reasonably screened and processed to accurately and efficiently estimate the scene surface points, thereby constructing a refined scene geometry model and significantly improving the accuracy in the construction of the scene geometry model.

[0020] 4. In the present invention, innovatively, the total loss of the neural network part and the Gaussian part is utilized to jointly optimize the Gaussian basis element attributes and the neural network parameters, promoting the simultaneous improvement of the scene geometry and the rendering quality, breaking through the limitation of the traditional method that focuses on one at the expense of the other, and achieving a more comprehensive optimization effect. Detailed implementation manners

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention.

[0022] Therefore, based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0023] Embodiment 1 This embodiment provides a high-precision three-dimensional scene and object reconstruction method for high-quality reconstruction of a three-dimensional scene / object, which specifically includes the following steps: Step 1, calculate the external camera parameters; Calculate the external camera parameters according to the real images, where the real images include a real depth map and a real RGB map .

[0024] The specific method is as follows: Step 1-1, read the first frame of real image, and use the internal camera parameters to back-project the image 2D points into the camera coordinate system to obtain the 3D point cloud data of the first frame : ; Among them, represents the first frame of real depth map, and represents the first frame of real RGB map.

[0025] Step 1-2, then detect the plane in the 3D point cloud of the first frame , calculate the point closest to the plane in the point cloud of this frame and use it as the origin of the world coordinate system, and Taking it as the x-axis, Taking it as the z-axis (i.e., the normal vector of the plane), the cross product of the x-axis and the z-axis gives the y-axis, thus obtaining the world coordinate system.

[0026] Step 1-3: After having the world coordinate system, next, convert the point cloud of the first frame of the real image in the camera coordinate system into the representation in the world coordinate system, and appropriately adjust the boundary of the TSDF voxel volume according to the actual range of the point cloud, thereby completing the initialization of the TSDF voxel volume.

[0027] Step 1-4: Update the TSDF voxel volume using subsequent frames of real images (including RGB images and depth images) to obtain the external camera parameters.

[0028] In Step 1-4, when updating the TSDF voxel volume, the specific steps are as follows: Step 1-4-1: Read the real depth map, real RGB map of the current frame and the camera internal parameters and back-project them into the point cloud in the camera coordinate system; Step 1-4-2: Use the global camera pose and camera internal parameters of the previous frame to render the fused scene through the ray casting algorithm to generate the depth rendering map and RGB rendering map of the fused scene; Step 1-4-3: Obtain the point cloud of the previous frame based on the depth rendering map and RGB rendering map, and then use the ICP algorithm to calculate the point cloud of the previous frame and the transformation matrix between the current frame point cloud; Step 1-4-4: Finally, use the transformation matrix between frames and the global camera pose of the previous frame to multiply to obtain the global camera pose of the current frame; That is, where represents the global pose corresponding to the i-th frame camera, represents the global pose corresponding to the (i - 1)-th frame camera, represents the transformation matrix from the (i - 1)-th frame camera to the i-th frame camera.

[0029] Step 2: Determine the coordinates and colors of the sampling points closest to the scene surface; Perform ray casting on the pixels of each frame of the real RGB image, and input the coordinates of the sampling points closest to the scene surface within the upsampling range of the ray into the SDF MLP network, and the SDF MLP network outputs the SDF prediction value and feature vector of the sampling points closest to the scene surface; Input the direction of the ray, the coordinates of the sampling points closest to the scene surface Gradient of SDF prediction value The SDF prediction value gradient, and the feature vector are input into the Color MLP network. The Color MLP network predicts the radiance of the sampling points closest to the scene surface and obtains the colors of the sampling points closest to the scene surface.

[0030] Determine the coordinates and colors of the sampling points closest to the scene surface. The specific steps are as follows: Step 2-1: Determine the sampling points closest to the scene surface; Perform ray casting on (partial) pixels of each frame of the real RGB image, connect the camera optical center and the pixels in the frame to project rays outward. The direction is . The initialization range of the sampling points on the ray is determined according to the depth of the corresponding pixels in the frame. The depth of the k-th pixel in each frame is . Through the sampling range on the ray passing through the k-th pixel, find two sampling points that meet the following screening conditions 、 , and form a set with these sampling points . Find the points where the SDF is zero among the sampling points in the set . These are the sampling points on the scene surface.

[0031] The above screening conditions are: ; Among them, represents the SDF estimated value of the sampling point on the ray passing through the k-th pixel in the i-th frame image, represents the SDF estimated value of another sampling point on the ray passing through the k-th pixel in the i-th frame image.

[0032] When finding the points where the SDF is zero among the sampling points in the set , the specific method is as follows: Interpolate between adjacent sampling points in the set to obtain a set : ; ; Find all the interpolation points between the sampling points that meet the screening conditions in the set , and extract the interpolation point closest to the camera from the set . This is the sampling point on the scene surface: ; Among them, ​denotes the interpolation point passing through two sampling points on the k-th pixel ray in the i-th frame image, denotes the sampling point passing through the k-th pixel ray in the i-th frame image, denotes another sampling point passing through the k-th pixel ray in the i-th frame image, denotes the SDF estimated value of the sampling point passing through the k-th pixel ray in the i-th frame image, denotes the SDF estimated value of another sampling point passing through the k-th pixel ray in the i-th frame image.

[0033] Step 2-2, output the SDF prediction value and feature vector of the sampling point; The coordinates of the sampling point closest to the scene surface within the sampling range on the ray ( ) are input into the SDF MLP network (this SDF MLP network can use an existing network without creative labor). This SDF MLP network outputs the SDF prediction value and feature vector of the sampling point closest to the scene surface.

[0034] Step 2-3, output the color of the sampling point; The direction of the ray , the coordinates of the sampling point closest to the scene surface , the gradient of the corresponding SDF prediction value output by the SDF MLP network , and the feature vector output by the SDF MLP network are input into the Color MLP network (this Color MLP network can use an existing network without creative labor). The Color MLP network predicts the radiance of the sampling point closest to the scene surface, and the color of the sampling point closest to the scene surface is obtained through the above steps.

[0035] Step 3, parameter update and optimization; Use the coordinates of the sampling point closest to the scene surface obtained in Step 2 and the color to initialize each attribute of the Gaussian basis element, including the position , shape , opacity λ, spherical harmonic function ; Select the Gaussian basis elements within the corresponding camera frustum (to save computational effort), and project the Gaussian basis elements within this frustum onto the camera's imaging plane based on the camera's internal parameters and the camera's external parameters calculated in Step 1.

[0036] Based on the rasterization technology of image patches (this is an existing technology), for the Gaussian basis elements projected into the pixel space, according to the opacity λ of each Gaussian basis element and its corresponding distribution weight, perform weighted summation on the pixels; Use the spherical harmonic function Color model the Gaussian basis elements, and generate the final rendered image by mixing the opacity λ and color distribution of the Gaussian basis elements. The rendered image includes RGB renderings, depth renderings, and normal renderings.

[0037] Using the rendered image, the real image, and the total loss , update and optimize the positions , shapes , opacity λ, spherical harmonic functions attributes, and the parameters of the SDF MLP network and Color MLP network.

[0038] The total loss is: .

[0039] Among them, the SDF prediction loss is: .

[0040] The SDF gradient consistency loss is: .

[0041] The RGB image loss is: .

[0042] The depth image loss is: .

[0043] The normal image loss is: .

[0044] Among them, all represent weight values; represents the SDF prediction loss, which is used to constrain the SDF prediction value of the sampling points output by the SDF MLP network, represents the coordinates of the upsampling points on the light ray passing through the k-th pixel in the i-th frame of the image, represents the set of sampling points, represents the number of upsampling points on the light ray passing through the k-th pixel in the i-th frame of the image, represents the SDF estimated value of the upsampling points on the light ray passing through the k-th pixel in the i-th frame of the image; represents the SDF gradient consistency loss, which is used to constrain the gradient consistency of the SDF, represents the gradient operator; Denotes the RGB image loss, which is used for the consistency constraint between the real RGB image and the RGB rendered image. Denotes the number of images. Denotes the i-th frame of the real RGB image. Denotes the i-th frame of the RGB rendered image. Denotes the depth image loss, which is used for the consistency constraint between the real depth image and the depth rendered image. Denotes the i-th frame of the real depth image. Denotes the i-th frame of the depth rendered image. Denotes the normal image loss, which is used to constrain the normal consistency between the normal map calculated from the depth map by computing the gradient and the normal rendered image. Denotes the gradient of the i-th frame of the real depth image. Denotes the i-th frame of the normal rendered image.

[0045] Step 4: Generate the scene geometry model and the rendered image. After the aforementioned training and optimization, the SDF MLP network, the optimal weights and bias values of the SDF MLP network, and the optimal Gaussian basis element attribute values have been obtained. On this basis, first initialize the parameters, including the grid resolution, the grid bounding box, divide the bounding box into several sub-grids according to the resolution, and generate grid points.

[0046] Traverse the sub-grids, generate grid points, layer by layer input the grid points into the optimized SDF MLP network. The SDF MLP network outputs the SDF prediction values of the grid points, and filters out the points whose SDF prediction values are close to 0 according to the SDF prediction values of the grid points; Use the Marching cubes algorithm (existing algorithm) to extract the grid surface from the SDF prediction values of the grid points to obtain a high-precision scene geometry model.

[0047] Input the internal parameters, external parameters of the camera corresponding to the perspective, and the attributes of the optimized Gaussian basis element, retain all the Gaussian basis elements within the camera frustum corresponding to the perspective, and project the Gaussian basis elements within the frustum onto the imaging plane of the camera; Based on the rasterization technology of image patches (existing technology), for the Gaussian basis elements projected into the pixel space, according to the opacity λ and the distribution weight of each optimized Gaussian basis element, perform weighted summation on the pixels; Use the optimized spherical harmonic function Perform color modeling on the Gaussian basis elements, and obtain a new perspective rendered image with high fidelity by mixing the opacity and color distribution of the Gaussian basis elements.

[0048] Finally, a high-precision scene geometry model with the same advantages as the implicit three-dimensional scene representation method and a new perspective image with high-fidelity rendering quality with the same advantages as the explicit three-dimensional scene representation method are obtained, thus effectively solving the technical problem that it is impossible to have both geometric accuracy and rendering quality in the three-dimensional reconstruction process.

[0049] Example 2 This example provides a high-precision three-dimensional scene and object reconstruction system for high-quality reconstruction of three-dimensional scenes / objects, which specifically includes: An external camera parameter calculation module for calculating external camera parameters based on real images, where the real images include real depth maps and real RGB maps .

[0050] The specific method is as follows: Step 1-1: Read the first frame of real image and use the internal camera parameters to back-project the 2D image points into the camera coordinate system to obtain the 3D point cloud data of the first frame : ; Among them, represents the first frame of real depth map, represents the first frame of real RGB map.

[0051] Step 1-2: Subsequently, detect the plane in the 3D point cloud of the first frame , calculate the point closest to the plane in the point cloud of this frame and use it as the origin of the world coordinate system, and use as the x-axis, and use as the z-axis (i.e., the normal vector of the plane), and the cross product of the x-axis and the z-axis to obtain the y-axis, thereby obtaining the world coordinate system.

[0052] After obtaining the world coordinate system, next, convert the point cloud of the first frame of real image in the camera coordinate system to be represented by the world coordinate system, and appropriately adjust the boundary of the TSDF voxel volume according to the actual range of the point cloud, thereby completing the initialization of the TSDF voxel volume.

[0053] Step 1-4: Update the TSDF voxel volume using subsequent frames of real images (including RGB images and depth images) to obtain the external camera parameters.

[0054] In Step 1-4, when updating the TSDF voxel volume, the specific steps are as follows: Step 1-4-1: Read the real depth map, real RGB map and internal camera parameters of the current frame and back-project them into the point cloud in the camera coordinate system; Step 1-4-2: Use the global camera pose and internal camera parameters of the previous frame to render the fused scene through the ray casting algorithm to generate a depth rendering map and an RGB rendering map of the fused scene; Step 1-4-3: Obtain the point cloud of the previous frame based on the depth rendering map and the RGB rendering map, and then use the ICP algorithm to calculate the point cloud of the previous frame and the point cloud of the current frame Transformation matrix between ; Step 1-4-4, finally using the transformation matrix between frames and the global camera pose of the previous frame Multiply to obtain the global camera pose of the current frame ; That is, ; Among them, Represents the global pose corresponding to the camera of the i-th frame, Represents the global pose corresponding to the camera of the (i-1)-th frame, Represents the transformation matrix from the camera of the (i-1)-th frame to the camera of the i-th frame.

[0055] The coordinate and color determination module of the sampling point closest to the scene surface is used to perform ray casting on the pixels of each frame of the real RGB image, and input the coordinates of the sampling point closest to the scene surface within the sampling range of the ray Into the SDF MLP network, and the SDF MLP network outputs the SDF prediction value and feature vector of the sampling point closest to the scene surface; Input the direction of the ray , the coordinates of the sampling point closest to the scene surface[[ID=3�]] , the gradient of the SDF prediction value , and the feature vector into the Color MLP network, and the Color MLP network predicts the radiance of the sampling point closest to the scene surface and obtains the color of the sampling point closest to the scene surface.

[0056] Determine the coordinates and colors of the sampling points closest to the scene surface, and the specific steps are as follows: Step 2-1, determine the sampling point closest to the scene surface; Perform ray casting on (part of) the pixels of each frame of the real RGB image, connect the camera optical center And project rays outward from the pixels in the frame, and its direction is . The initial sampling range of the sampling points on the ray is determined according to the depth of the corresponding pixels in the frame. The depth of the k-th pixel in each frame is , and through the sampling range Of the ray passing through the k-th pixel, find two sampling points that meet the following screening conditions , , and form a set of these sampling points , find the point where the SDF is zero between the sampling points in the set , which is the sampling point on the scene surface.

[0057] The above screening conditions are: ; Among them, represents the SDF estimated value of the upsampling point on the k-th pixel ray in the i-th frame image, represents the SDF estimated value of another sampling point on the k-th pixel ray in the i-th frame image.

[0058] When finding the point where the SDF is zero among the sampling points in the set , the specific method is as follows: Interpolate between two adjacent sampling points in the set to obtain the set : ; ; Find all the interpolation points between the sampling points that meet the screening conditions in the set , and extract the interpolation point closest to the camera from the set as the sampling point on the scene surface: ; ; Among them, represents the interpolation point of two sampling points on the k-th pixel ray in the i-th frame image, represents the sampling point on the k-th pixel ray in the i-th frame image, represents another sampling point on the k-th pixel ray in the i-th frame image, represents the SDF estimated value of the sampling point on the k-th pixel ray in the i-th frame image, represents the SDF estimated value of another sampling point on the k-th pixel ray in the i-th frame image.

[0059] Step 2-2, output the SDF prediction value and feature vector of the sampling point; Input the coordinates ( ) of the sampling point closest to the scene surface within the upsampling range of the ray into the SDF MLP network (this SDF MLP network can use an existing network without creative effort), and this SDF MLP network outputs the SDF prediction value and feature vector of the sampling point closest to the scene surface.

[0060] Step 2-3, output the color of the sampling point; Input the direction of the ray, the coordinates of the sampling point closest to the scene surface, and the gradient The feature vectors output by the SDF MLP network are input into the Color MLP network (an existing network can be used for this Color MLP network without creative effort). The Color MLP network predicts the radiance closest to the sampled points on the scene surface. Through the above steps, the color of the sampled points closest to the scene surface is obtained.

[0061] The parameter update and optimization module is used to determine the coordinates of the sampled points closest to the scene surface obtained by the coordinate and color determination module of the sampled points closest to the scene surface and each attribute of the color initialization Gaussian basis element, including the position 、shape 、opacity λ, spherical harmonic function ; Select the Gaussian basis elements within the corresponding camera frustum (to save computational effort). Based on the camera internal parameters and the camera external parameters calculated in step 1, project the Gaussian basis elements within the frustum onto the imaging plane of the camera.

[0062] Based on the rasterization technology of image patches (this is an existing technology), for the Gaussian basis elements projected into the pixel space, according to the opacity λ of each Gaussian basis element and its corresponding distribution weight, perform weighted summation on the pixels; Use the spherical harmonic function to perform color modeling on the Gaussian basis elements, and generate the final rendered image by mixing the opacity λ and color distribution of the Gaussian basis elements. The rendered image includes an RGB rendered image, a depth rendered image, and a normal rendered image.

[0063] Utilize the rendered image, the real image, and the total loss to update and optimize the position 、shape 、opacity λ, spherical harmonic function attributes, as well as the parameters of the SDF MLP network and the Color MLP network.

[0064] The total loss is: .

[0065] Among them, the SDF prediction loss is: .

[0066] The SDF gradient consistency loss is: .

[0067] The RGB image loss is: .

[0068] Depth Image Loss for: .

[0069] Normal image loss for: .

[0070] in, All represent weight values; Represents the SDF prediction loss, which is used to constrain the SDF prediction value of the SDFMLP network output sampling point. Represents the coordinates of the sampling point on the ray passing through the k-th pixel in the i-th frame image, represents a set of sampling points, Indicates the number of sampling points on the ray passing through the k-th pixel in the i-th frame image, Represents the estimated SDF value of the sampling point on the ray passing through the k-th pixel in the i-th frame image; Represents the SDF gradient consistency loss, which is used to constrain the gradient consistency of SDF. represents the gradient operator; Represents the RGB image loss, which is used to constrain the consistency between the real RGB image and the RGB rendering image. Indicates the number of images, Represents the real RGB image of the i-th frame, Represents the RGB rendering of the i-th frame; Represents the depth image loss, which is used to constrain the consistency between the real depth map and the depth rendering map. represents the true depth map of the i-th frame, Represents the depth rendering image of the i-th frame; Represents the normal image loss, which is used to calculate the gradient from the depth map to obtain the normal consistency between the normal map and the normal rendering map. represents the gradient of the true depth map of the i-th frame, Represents the normal rendering image of the i-th frame.

[0071] The scene geometry model and rendering generation module is used to input grid points layer by layer into the optimized SDF MLP network. The SDF MLP network outputs the SDF prediction values of the grid points and filters out points with SDF prediction values close to 0 based on the SDF prediction values of the grid points. The Marching cubes algorithm is used to extract the grid surface from the SDF prediction values of the grid points to obtain the scene geometry model. Input the intrinsic and extrinsic parameters of the camera corresponding to the perspective and the attributes of the optimized Gaussian basis elements, retain all the Gaussian basis elements within the camera frustum corresponding to the perspective, and project the Gaussian basis elements within the frustum onto the imaging plane of the camera; for the Gaussian basis elements projected into the pixel space, perform weighted summation on the pixels according to the opacity λ and distribution weight of each optimized Gaussian basis element; use the optimized spherical harmonic function Perform color modeling on the Gaussian basis elements, and obtain a high-fidelity new perspective rendering by mixing the opacity and color distribution of the Gaussian basis elements.

[0072] Embodiment 3 A computer device includes a memory and a processor. When the computer program stored in the memory is executed by the processor, the processor is caused to execute the steps of the high-precision three-dimensional scene and object reconstruction method.

[0073] Among them, the computer device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device can perform human-computer interaction with the user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device, etc.

[0074] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or D interface display memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device. Of course, the memory may also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the memory is commonly used to store the operating system installed on the computer device and various application software, such as the program code of the high-precision three-dimensional scene and object reconstruction method. In addition, the memory can also be used to temporarily store various data that have been output or will be output.

[0075] In some embodiments, the processor may be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the high-precision three-dimensional scene and object reconstruction method.

[0076] Embodiment 4 A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor is caused to execute the steps of the high-precision three-dimensional scene and object reconstruction method.

[0077] Wherein, the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to cause the at least one processor to execute the steps of the high-precision three-dimensional scene and object reconstruction method as described above.

[0078] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the high-precision three-dimensional scene and object reconstruction method described in the embodiments of the present application.

Claims

1. A high-precision three-dimensional scene and object reconstruction method, characterized in that, Including the following steps: Step 1, calculate the external parameters of the camera; Calculating the extrinsic camera parameters based on real images, where the real images include a real depth map and a real RGB image ; Step 2, determine the coordinates and colors of the sampling points closest to the surface of the scene; Perform ray casting on the pixels of each frame of the real RGB image, and input the coordinates of the sampling points closest to the scene surface within the upsampling range of the ray into the SDF MLP network, which outputs the SDF prediction value and the feature vector of the sampling point closest to the scene surface; The direction of the light , the coordinates of the sampling point closest to the surface of the scene , the gradient of the SDF prediction value , and the feature vector are input into the Color MLP network. The Color MLP network predicts the radiance of the sampling point closest to the surface of the scene and obtains the color of the sampling point closest to the surface of the scene; Step 3, update and optimize the parameters; Using the coordinates of the sampling points closest to the surface of the scene and the color to initialize the position of the Gaussian basis element , shape , opacity λ, spherical harmonics , select the Gaussian basis elements within the corresponding camera frustum, and project the Gaussian basis elements within the frustum onto the imaging plane of the camera based on the camera internal parameters and the camera external parameters calculated in step 1; For the Gaussian basis elements projected onto the pixel space, weighted summation of pixels is performed according to the opacity λ and distribution weights of each Gaussian basis element; the spherical harmonic function is used Color modeling of the Gaussian basis elements is carried out, and the final rendered image is generated by mixing the opacity and color distribution of the Gaussian basis elements. The rendered image includes RGB rendered images, depth rendered images, and normal rendered images; Using the rendered image and the real image to update and optimize the position , shape , opacity λ, spherical harmonic function attributes, as well as the parameters of the SDF MLP network and the Color MLP network; Step 4, generate the scene geometric model and the rendering; Input the grid points layer by layer into the optimized SDF MLP network. The SDF MLP network outputs the SDF prediction values of the grid points, and filter out the points with SDF prediction values close to 0 according to the SDF prediction values of the grid points; Use the Marching cubes algorithm to extract the grid surface from the SDF prediction values of the grid points to obtain the scene geometric model; Input the intrinsic and extrinsic parameters of the camera corresponding to the perspective and the attributes of the optimized Gaussian basis elements. Retain all the Gaussian basis elements within the camera frustum corresponding to the perspective, and project the Gaussian basis elements within this frustum onto the imaging plane of the camera. For the Gaussian basis elements projected into the pixel space, perform a weighted sum of the pixels according to the opacity λ and the distribution weight of each optimized Gaussian basis element. Use the optimized spherical harmonics Perform color modeling on the Gaussian basis elements. By mixing the opacity and color distribution of the Gaussian basis elements, obtain a new perspective rendering with high fidelity.

2. The high-precision three-dimensional scene and object reconstruction method according to claim 1, wherein When calculating the external parameters of the camera in Step 1, the specific method is: Step 1-1, read the first frame of real image and use the camera intrinsic parameters Back-project the image into the camera coordinate system to obtain the 3D point cloud data of the first frame : ； Among them, represents the first-frame real depth map, represents the first-frame real RGB map; Step 1-2, detect the plane in the first-frame 3D point cloud , calculate the point closest to the plane in the point cloud of this frame as the origin of the world coordinate system, and take as the x-axis, take as the z-axis, and obtain the y-axis by taking the cross product of the x-axis and the z-axis to get the world coordinate system; Step 1-3, convert the point cloud of the first frame of real image in the camera coordinate system into the world coordinate system, and adjust the boundary of the TSDF voxel volume according to the actual range of the point cloud to complete the initialization of the TSDF voxel volume; Step 1-4, update the TSDF voxel volume using subsequent frames of real images to obtain the external parameters of the camera.

3. A high-precision three-dimensional scene and object reconstruction method according to claim 2, characterized in that, In Step 1-4, the specific steps for updating the TSDF voxel volume are: Read the real depth map, real RGB map of the current frame and the camera intrinsics, and back-project them into the point cloud in the camera coordinate system. Use the global camera pose and camera intrinsics of the previous frame to render the fused scene through the ray casting algorithm, generating the depth rendering map and RGB rendering map of the fused scene. Obtain the point cloud of the previous frame based on the depth rendering map and RGB rendering map, and then use the ICP algorithm to calculate the point cloud of the previous frame and the point cloud of the current frame to obtain the transformation matrix between them. Finally, use the transformation matrix between frames and the global camera pose of the previous frame to multiply and obtain the global camera pose of the current frame .

4. The high-precision three-dimensional scene and object reconstruction method according to claim 1, characterized in that In Step 2, the specific method for determining the sampling points on the scene surface is: Perform ray casting on the pixels of each frame of the real RGB image, and connect the camera optical center and project rays outward from the pixels in the frame, with the direction being ; Find two sampling points on the ray that satisfy the following screening conditions , , and form a set with these sampling points Find in the set a point where the SDF is zero between the sampling points, which is the sampling point on the scene surface; The screening conditions are: ； wherein, represents the SDF estimation value of the upsampling point on the k-th pixel ray in the i-th frame image, represents the SDF estimation value of another sampling point on the k-th pixel ray in the i-th frame image.

5. The high-precision three-dimensional scene and object reconstruction method according to claim 4, characterized in that, Finding a set When there are points with zero SDF between the sampling points in [the set], the specific method is as follows: Interpolate between adjacent sampling points in the set to obtain the set : ； ； Find the interpolation points between all sampling points that meet the screening criteria in the set , and extract the interpolation point closest to the camera from the set . This is the sampling point on the scene surface: ​ ； Among them, represents the interpolation point of two sampling points on the k-th pixel ray in the i-th frame of image, represents the sampling point on the k-th pixel ray in the i-th frame of image, represents the other sampling point on the k-th pixel ray in the i-th frame of image, represents the SDF estimated value of the sampling point on the k-th pixel ray in the i-th frame of image, represents the SDF estimated value of the other sampling point on the k-th pixel ray in the i-th frame of image.

6. The high-precision three-dimensional scene and object reconstruction method according to claim 1, characterized in that In step 3 when optimizing the parameters, the total loss is as follows: ； ； ； ； ； ； Among them, both represent weight values; represents the SDF prediction loss, which is used to constrain the SDF prediction value of the sampling points output by the SDF MLP network. represents the coordinates of the sampling points on the k-th pixel ray in the i-th frame of the image. represents the set of sampling points. represents the number of sampling points on the k-th pixel ray in the i-th frame of the image. represents the SDF estimated value of the sampling points on the k-th pixel ray in the i-th frame of the image. represents the SDF gradient consistency loss, which is used to constrain the gradient consistency of the SDF. represents the gradient operator. represents the RGB image loss, which is used for the consistency constraint between the real RGB image and the RGB rendered image. represents the number of images. represents the i-th frame of the real RGB image. represents the i-th frame of the RGB rendered image. represents the depth image loss, which is used for the consistency constraint between the real depth image and the depth rendered image. represents the i-th frame of the real depth image. represents the i-th frame of the depth rendered image. represents the normal image loss, which is used to constrain the normal consistency between the normal map calculated from the depth map and the normal rendered image. represents the gradient of the i-th frame of the real depth image. represents the i-th frame of the normal rendered image.

7. A high-precision three-dimensional scene and object reconstruction system, characterized in that, Including: The external camera parameter calculation module is used to calculate the external camera parameters based on real images, where the real images include real depth maps and real RGB maps ; The coordinate and color determination module for the sampling point closest to the scene surface is used to perform ray casting on the pixels of each frame of the real RGB image, and input the coordinates of the sampling point closest to the scene surface within the sampling range of the ray into the SDF MLP network, and the SDF MLP network outputs the SDF prediction value and the feature vector of the sampling point closest to the scene surface; The direction of the light , the coordinates of the sampling point closest to the surface of the scene , the gradient of the SDF prediction value , and the feature vector are input into the Color MLP network, which predicts the radiance of the sampling point closest to the surface of the scene and obtains the color of the sampling point closest to the surface of the scene; The parameter update and optimization module is used to initialize the position of the Gaussian basis element by using the coordinates and colors of the sampling points closest to the surface of the scene. and shape , opacity λ, spherical harmonic function , select the Gaussian basis elements within the corresponding camera frustum, and project the Gaussian basis elements within the frustum onto the imaging plane of the camera based on the internal parameters of the camera and the external parameters of the camera calculated by the camera external parameter calculation module. For the Gaussian basis elements projected onto the pixel space, weighted summation is performed on the pixels according to the opacity λ and distribution weights of each Gaussian basis element; the spherical harmonic function is used Color modeling is performed on the Gaussian basis elements, and the final rendered image is generated by mixing the opacity and color distribution of the Gaussian basis elements. The rendered image includes an RGB rendered image, a depth rendered image, and a normal rendered image; Position of the Gaussian basis element using the rendered image and the real image , shape , opacity λ, spherical harmonic function attributes, and the parameters of the SDF MLP network and the Color MLP network are updated and optimized; A scene geometric model and rendering generation module, which is used to input grid points layer by layer into the optimized SDF MLP network. The SDF MLP network outputs the SDF prediction values of the grid points, and filter out the points with SDF prediction values close to 0 according to the SDF prediction values of the grid points; Use the Marching cubes algorithm to extract the grid surface from the SDF prediction values of the grid points to obtain the scene geometric model; Input the intrinsic and extrinsic parameters of the camera corresponding to the perspective and the attributes of the optimized Gaussian basis elements, retain all the Gaussian basis elements within the camera frustum corresponding to the perspective, and project the Gaussian basis elements within the frustum onto the imaging plane of the camera; for the Gaussian basis elements projected into the pixel space, perform weighted summation on the pixels according to the opacity λ and the distribution weight of each optimized Gaussian basis element; use the optimized spherical harmonics Perform color modeling on the Gaussian basis elements, and obtain a new perspective rendering with high fidelity by mixing the opacity and color distribution of the Gaussian basis elements.

8. A computer device, characterized in that: Including a memory and a processor, the memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 6.

Images