Three-dimensional scene real-time reconstruction method fusing symbol distance field and three-dimensional Gaussian representation

Abstract

The invention discloses a three-dimensional scene real-time reconstruction method fusing a symbol distance field and three-dimensional Gaussian representation. The method comprises the steps of firstly receiving a color-depth frame captured by a camera, obtaining a camera pose of a current frame through camera tracking estimation, then fusing and updating color and depth information of the current frame and a global symbol distance field, adaptively adding, optimizing and removing three-dimensional gauss, and finally correcting the color of a symbol distance field rendering result, so as to obtain a target image. And the reconstruction reality sense is improved. According to the invention, through fusion of three-dimensional Gaussian and symbol distance field representation, high-quality scene reconstruction can be realized by using a symbol distance field and a small amount of three-dimensional Gaussian, ultra-high-speed real-time reconstruction of a three-dimensional scene is realized, and high-precision geometry and high-quality textures are kept at the same time.

Description

Technical Field

[0001] This invention relates to the field of 3D scene reconstruction, specifically to a real-time 3D scene reconstruction method that integrates symbolic distance field and 3D Gaussian representation. Background Technology

[0002] Real-time large-scale 3D reconstruction has been a long-standing research problem in computer graphics and vision, crucial in many applications, including virtual reality / augmented reality, autonomous robots, and interactive scanning with real-time feedback. With the widespread adoption of 3D Gaussian splashing technology and RGBD cameras (such as Microsoft Kinect and Intel RealSense), various SLAM methods (simultaneous localization and mapping) using 3D Gaussian as the map representation have been proposed. For example, RTG-SLAM achieves a reconstruction speed of around 15 FPS by reducing the number of Gaussians in the scene through a compact Gaussian representation, while GauS SLAM uses 2D Gaussians to improve the geometric accuracy of the reconstruction and camera tracking accuracy. While these methods achieve high-quality reconstruction results, their reconstruction speed is still significantly slower than traditional geometric SLAM methods. Although the state-of-the-art GS-ICP SLAM claims to achieve real-time reconstruction at 100 FPS, its actual mapping speed is much slower than its tracking speed, and the system may terminate before fully completing mapping, resulting in poor reconstruction quality. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention discloses a real-time reconstruction method for 3D scenes that integrates symbolic distance field and 3D Gaussian representation. By integrating 3D Gaussian and symbolic distance field representations, this method can achieve high-quality scene reconstruction using symbolic distance field and a small amount of 3D Gaussian representation, realizing ultra-high-speed real-time reconstruction of 3D scenes while maintaining high-precision geometry and high-quality texture.

[0004] This invention is achieved through the following technical solution:

[0005] This invention first provides a method for real-time reconstruction of 3D scenes by fusing symbolic distance fields and 3D Gaussian representation, including the following steps:

[0006] (1) The symbolic distance field of the global map is rendered into depth and color images under the view of each frame captured by the RGBD camera; the depth image of the current frame is matched and registered with the depth image of the global map rendered under the view of the previous frame, and the camera pose is estimated and the key frame sequence is updated.

[0007] (2) Fuse the color information and depth information of the current frame image into the symbolic distance field of the global map, and update the properties of the visible voxels in the symbolic distance field under the current view.

[0008] (3) Add a new three-dimensional Gaussian to the three-dimensional Gaussian of the global map, select an optimized view from the historical view and render the global map; the rendering process is a weighted average of the rendering results of the three-dimensional Gaussian of the global map and the rendering results of the symbolic distance field of the global map.

[0009] (4) Establish and optimize the rendering error function, update the relevant parameters of the 3D Gaussian in the global map; delete redundant Gaussian in the global map; and finally complete the real-time reconstruction of the 3D scene. The error between the rendering result after weighted averaging of the rendering error function and the color image of the current frame.

[0010] Furthermore, step (1) specifically includes the following sub-steps:

[0011] (1.1) Image data acquisition and preprocessing: During the scanning process of the RGBD camera, the color image and depth image are acquired in the current frame; after the image is transmitted to the computing unit, the depth map is back-projected according to the camera calibration intrinsic parameters to obtain the vertex map and calculate the normal map; the normal vector of each vertex on the vertex map is calculated based on the cross product of the coordinates of the surrounding pixel vertices;

[0012] (1.2) Image rendering of the symbolic distance field of the global map: The global map maintains a voxel-represented symbolic distance field as one of the basic building blocks of the map; specifically, the vicinity of the scene surface is divided into a series of small cubes with fixed side lengths, called voxels, and a spatial voxel is returned for any three-dimensional spatial point near the surface; each voxel contains the following attributes: color, symbolic distance, and update count, where the absolute value of the symbolic distance represents the distance from the voxel center to the scene surface, a positive value indicates that the voxel is outside the scene surface, and a negative value indicates that the voxel is inside the scene surface; during the rendering of the symbolic distance field, given any camera pose and camera intrinsic parameters, the zero-value surface in the symbolic distance field corresponding to each pixel is obtained through ray casting technology; the three-dimensional coordinates of the zero-value surface are projected onto the camera coordinate system to obtain the depth of the current pixel, and the three-dimensional coordinates of the zero-value surface are interpolated with the colors of nearby voxels to obtain the chroma of the current pixel;

[0013] (1.3) Camera pose estimation: Based on the model rendering results and the camera intrinsic parameters and camera pose of the previous frame, the 3D coordinates and normal vectors of each vertex in the depth map of the previous frame are calculated using the same calculation method as the current frame to obtain the vertex map and normal map, which form the point cloud under the view of the previous frame; at the same time, the vertex map and normal map of the current frame form the point cloud under the view of the current frame; based on the correspondence between pixels, the correspondence between point clouds is established, and the point-plane iterative nearest point algorithm is used to estimate the camera pose matrix;

[0014] (1.4) Maintaining the keyframe sequence: The first frame at the start of the scan is fixed as the keyframe. If the spatial distance between the pose of the current frame and the previous keyframe changes by more than 0.3 meters or the angle changes by more than 30 degrees, then this frame is marked as a new keyframe and added to the keyframe sequence.

[0015] Furthermore, in step (1.2), the chroma of the current pixel is obtained by interpolating the three-dimensional coordinates of the zero-value surface with the colors of nearby voxels. The calculation specifically includes the following steps:

[0016] (1.2.1) Given any camera pose T g Given the camera intrinsic parameter matrix K, any pixel u in the image corresponds to a ray r projected from the camera's optical center. u :

[0017]

[0018] Among them, R g The rotation matrix representing the camera pose; t g The translation matrix represents the camera pose, and θ is the distance parameter of the light rays;

[0019] (1.2.2) Proceed in steps along the ray direction with a certain step size δθ until, for a certain ray distance θ, the SDF(r) is satisfied. u (θ))>0 and SDF(r) u (θ+δθ))<0;

[0020] (1.2.3) Calculate the more accurate surface distance SDF for points on the light ray. + (r u (θ+δθ)) and SDF + (r u (θ)), SDF + The calculation formula is as follows:

[0021] SDF + (p)=Triliner_Interpolate({SDF(p i )}), i = 0, 1, ..., 7;

[0022] Among them, {SDF(p i )} represents the sign distances between the eight adjacent voxels of p, and Triliner_Interpolate represents trilinear interpolation based on the distance from p to the center of each voxel;

[0023] (1.2.4) Calculate the zero surface r u (θ * The zero-value surface is then projected onto the camera coordinate system to obtain the rendering depth.

[0024] (1.2.5) Calculate the zero surface r u (θ * The corresponding color is used as the rendering color.

[0025] c + (p)=Triliner_Interpolate({c(p i )}), i = 0, 1, ..., 7;

[0026] The calculation method is the same as step (1.2.3).

[0027] Furthermore, step (2) includes the following sub-steps:

[0028] (2.1) Voxel mesh projection: For all voxels within the visible range of the current frame, project the voxel centers into the image space to obtain pixel coordinates;

[0029] (2.2) Calculate the symbolic distance of the current frame: For each voxel, calculate the symbolic distance of the current frame based on the depth map of the current frame;

[0030] (2.3) Weighted update result: With the weight of the new frame as 1, update the symbolic distance, color, and update count attributes within each voxel.

[0031] Furthermore, step (3) specifically includes the following sub-steps:

[0032] (3.1) The global map also maintains some 3D Gaussians as one of the basic building blocks of the map. Each 3D Gaussian includes attributes such as opacity, spatial position, scaling parameters, rotation parameters, and spherical harmonic coefficients. During the rendering of the 3D Gaussian, given any camera pose and camera intrinsic parameters, the 3D Gaussian is transformed into a 2D Gaussian on the image plane through Gaussian splashing technology. The Gaussians intersecting each pixel are recorded and culled based on the rendering depth of the signed distance field. For color rendering, the RGB values ​​of each intersecting Gaussian for each pixel are accumulated according to the opacity to obtain the chroma of the current pixel. For weighted rendering, the opacity of each intersecting Gaussian for each pixel is accumulated to obtain the weight of the current pixel. The color rendering results of the signed distance field and the color rendering results of the 3D Gaussian are weighted and averaged to obtain the rendering result of the global gradient.

[0033] (3.2) Compare the rendered image of the global map in the current frame with the image acquired in the current frame. Areas with large color errors are areas that need to have Gaussian added.

[0034] (3.3) Sample pixels with a set threshold in the area where Gaussian needs to be added to add new Gaussian;

[0035] (3.4) Randomly select a given number of frames from the keyframe sequence and uniformly select a given number of frames from the most recent ordinary frames to obtain the camera viewpoint and corresponding color images of these frames;

[0036] (3.5) Render the global map under these selected viewpoints, establish a rendering error function and iteratively optimize it;

[0037] (3.6) Delete the three-dimensional Gaussian in the global map with a scale smaller than 0.003, a scale greater than 0.1, and an opacity less than 0.005.

[0038] Further, in step (3.1), the color rendering results of the symbolic distance field and the color rendering results of the three-dimensional Gaussian are weighted and averaged to obtain the global gradient rendering result; specifically:

[0039] (3.1.1) Given any camera pose T g Given the camera intrinsic parameter matrix K, any pixel u in the image corresponds to a ray r(u) projected from the camera optical center;

[0040] (3.1.2) The color of ray r(u) is calculated by weighted accumulation and depth culling techniques. and weight Complete the rendering of the color image and the weighted image:

[0041]

[0042] Where, d i This represents the depth of the Gaussian plane in the camera coordinate system. This represents the depth obtained from the symbolic distance field rendering; ∈ is a small positive value used to prevent erroneous culling. It is an indicator function. This means removing all depth values ​​greater than 1. Gauss, c i According to SH i The Gaussian color obtained from the observation angle, f i After being projected into pixel space by Gaussian splashing, it is based on the Gaussian center μ i The covariance matrix Σ 2D,i The resulting opacity;

[0043] (3.1.3) Color rendering results of 3D Gaussian Rendering results of the symbolic distance field The data is then blended to obtain the rendered global map:

[0044]

[0045] The above expression treats the weighting of the symbolic distance field as 1, and the weighting of the three-dimensional Gaussian field as... Perform a weighted average.

[0046] Furthermore, step (3.4) specifically includes the following sub-steps:

[0047] (3.4.1) Randomly select a camera view obtained in step (3.3), and simultaneously render the symbolic distance field and the three-dimensional Gaussian in the global map. The color rendering results of the symbolic distance field and the color rendering results of the three-dimensional Gaussian are weighted and averaged. The weight of the rendering result of the symbolic distance field is 1, and the weight of the color rendering result of the three-dimensional Gaussian is the weight map it renders.

[0048] (3.4.2) Calculate the L1 distance between the weighted average rendering result and the chroma image corresponding to the selected viewpoint as the rendering error function;

[0049] (3.4.3) Calculate the gradient and update the 3D Gaussian in the global map using the Adam optimizer;

[0050] (3.4.4) Repeat steps (3.4.1) to (3.4.3) 20 times to complete the iterative optimization.

[0051] The present invention also provides an electronic device, including one or more processors, for implementing the aforementioned method for real-time reconstruction of a 3D scene by fusing a symbolic distance field and a 3D Gaussian representation.

[0052] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for real-time reconstruction of a 3D scene by fusing a symbolic distance field and a 3D Gaussian representation.

[0053] The beneficial effects of this invention are as follows:

[0054] This invention presents a real-time 3D scene reconstruction method that integrates symbolic distance field and 3D Gaussian representation. It estimates the camera pose of the current frame in real time from a synchronized color-depth image video stream acquired from the camera. The method uses a symbolic distance field to represent the 3D structure of the scene and 3D Gaussian representation to supplement color residuals. During scanning, the symbolic distance field is updated in real time, and 3D Gaussian parameters are adaptively added. Simultaneously, the 3D Gaussian parameters are optimized based on chroma image information. This invention can describe a 3D scene using a symbolic distance field and a small amount of 3D Gaussian representation, significantly reducing the optimization time for 3D Gaussian. It achieves ultra-high-speed reconstruction while maintaining the geometric and textural accuracy of the reconstructed results. Attached Figure Description

[0055] Figure 1 This is a flowchart of the method of the present invention;

[0056] Figure 2 This is a schematic diagram of the rendering process of the present invention;

[0057] Figure 3 This is a comparison chart of the results of the present invention. Detailed Implementation

[0058] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0059] The core of this invention lies in reconstructing the symbolic distance field and the three-dimensional Gaussian representation of the scene using color image / depth image streams. During the scanning process of the RGBD camera, a global map that fuses the symbolic distance field and the three-dimensional Gaussian representation can be established to reconstruct the three-dimensional scene while estimating the camera pose in real time. The reconstruction result supports high-fidelity real-time rendering.

[0060] like Figure 1 and Figure 2 As shown, the real-time reconstruction method for 3D scenes by fusing symbolic distance field and 3D Gaussian representation of the present invention includes the following steps:

[0061] Step 1: The symbolic distance field of the global map is rendered into depth and color images from the viewpoint of each frame captured by the RGBD camera; the depth image of the current frame captured by the RGBD camera is matched and registered with the depth image of the global map rendered from the viewpoint of the previous frame to estimate the camera pose; the keyframe sequence is updated. Step 1 consists of the following sub-steps:

[0062] 1.1 Image Data Acquisition and Preprocessing:

[0063] During the RGBD camera scanning process, a color image C is acquired in the current frame. k (u) and depth image D k (u). After the image is transmitted to the computing unit, the depth map is back-projected according to the camera calibration intrinsic parameters to obtain the vertex map and calculate the normal map.

[0064] The 3D vertex coordinates V of each pixel in the depth map k The calculation formula for (u) is as follows:

[0065] V k (u)=D k (u)K -1 u;

[0066] Where u represents the two-dimensional coordinate matrix of the pixel, u = (i,j), and K represents the camera intrinsic parameter matrix;

[0067] The normal vector of each vertex in the vertex graph is calculated based on the cross product of the coordinates of its surrounding pixels, as shown in the following formula:

[0068]

[0069] 1.2 Image rendering of global map symbol distance field:

[0070] The global map maintains a voxelized symbolic distance field S as one of the basic building blocks of the map. Specifically, the vicinity of the scene surface is divided into a series of small cubes with fixed side lengths, called voxels. For any 3D spatial point p near the surface, S(p) returns a spatial voxel s. Each voxel contains the following attributes: color c(p), symbolic distance SDF(p), and update count n(p), where the absolute value of the symbolic distance represents the distance from the voxel center to the scene surface; a positive value indicates that the voxel is outside the scene surface, and a negative value indicates that the voxel is inside the scene surface.

[0071] During the rendering of the symbolic distance field, given arbitrary camera pose and camera intrinsic parameters, the zero-value surface in the symbolic distance field corresponding to each pixel is obtained through ray casting. The 3D coordinates of the zero-value surface are projected onto the camera coordinate system to obtain the depth of the current pixel. The 3D coordinates of the zero-value surface are then interpolated with the colors of nearby voxels to obtain the chroma of the current pixel. The specific calculation steps are as follows:

[0072] (1.2.1) Given any camera pose T g Given the camera intrinsic parameter matrix K, any pixel u in the image corresponds to a ray r projected from the camera's optical center. u :

[0073]

[0074] Among them, R g The rotation matrix representing the camera pose; t g The translation matrix represents the camera pose, and θ is the distance parameter of the light rays.

[0075] (1.2.2) Proceed in steps along the ray direction with a certain step size δθ until, for a certain ray distance θ, the SDF(r) is satisfied. u (θ))>0 and SDF(r) u (θ+δθ))<0;

[0076] (1.2.3) Calculate the more accurate surface distance SDF for points on the light ray. + (r u (θ+δθ)) and SDF + (r u (θ)), SDF + The calculation formula is as follows:

[0077] SDF + (p)=Triliner_Interpolate({SDF(p i )}), i = 0, 1, ..., 7;

[0078] Among them, {SDF(p i )} represents the symbol distances of the eight adjacent voxels of p, and Triliner_Interpolate represents trilinear interpolation based on the distance from p to the center of each voxel.

[0079] (1.2.4) Calculate the zero surface r u (θ * The zero-value surface is then projected onto the camera coordinate system to obtain the rendering depth.

[0080] (1.2.5) Calculate the zero surface r u (θ * The corresponding color is used as the rendering color.

[0081] c + (p)=Triliner_Interpolate({c(p i )}), i = 0, 1, ..., 7;

[0082] The calculation method is similar to (1.2.3).

[0083] 1.3 Camera Pose Estimation

[0084] Based on the model rendering results Using the camera intrinsics and pose from the previous frame, the 3D coordinates and normal vectors of each vertex in the depth map of the previous frame are calculated using the same method as in the current frame, resulting in a vertex map and a normal map, which together form the point cloud from the previous frame's viewpoint. Simultaneously, the vertex map and normal map of the current frame form the point cloud from the current frame's viewpoint. Based on the correspondence between pixels, the correspondence between point clouds is established, and the point-plane iterative nearest-point algorithm is used to estimate the camera pose matrix T. g,k .

[0085] 1.4 Maintaining the keyframe sequence

[0086] The first frame of the scan is fixed as a keyframe. If the spatial distance between the pose of the current frame and the previous keyframe changes by more than 0.3 meters or the angle changes by more than 30 degrees, then this frame is marked as a new keyframe and added to the keyframe sequence.

[0087] Step 2: Fuse the color and depth information of the current frame into the symbolic distance field of the global map, and update the properties of the visible voxels in the symbolic distance field from the current viewpoint. Step 2 includes the following sub-steps:

[0088] 2.1 Voxel Mesh Projection

[0089] For all voxels within the visible range of the current frame, project the voxel center o into the image space to obtain the pixel coordinates u. o :

[0090]

[0091] Where floor represents flooring, and π represents dehomogenization. Specifically, for any three-dimensional vector p = (x, y, z), π(p) = q, satisfying

[0092] 2.2 Calculate the symbol distance of the current frame

[0093] For each voxel, calculate the symbolic distance (SDF) of the current frame based on the depth map of the current frame. k ′ :

[0094]

[0095] λ=||K -1 u o ||;

[0096] in D represents the vertical distance from the current voxel center to the camera. k (u o () indicates the measured depth from the scene surface to the camera.

[0097] 2.3 Weighted Update Results

[0098] With a weight of 1 for the new frame, update the symbolic distance, color, and update count attributes within each voxel:

[0099]

[0100] n k ←n k-1 +1;

[0101] SDF k-1 c k-1 n k-1 This indicates the result after the previous frame has been merged and updated.

[0102] Step 3: Add a new 3D Gaussian to the global map's 3D Gaussian; select an optimized viewpoint from the historical perspective and render the global map; establish and optimize a rendering error function; update the relevant parameters of the global 3D Gaussian; delete redundant Gaussians in the global map; the rendering process involves a weighted average of the rendering results of the global map's 3D Gaussian and the rendering results of the global map's signed distance field; the error between the weighted average rendering result and the chroma image of the current frame is determined by the rendering error function. Step 3 includes the following sub-steps:

[0103] 3.1 Hybrid Rendering of Signed Distance Field and 3D Gaussian

[0104] The global map maintains a set of three-dimensional Gaussian maps {G1, G2, ..., G...} m ,..,G n As the basic building block of a map, the m-th Gaussian sigma includes the spatial location x. m Opacity α m Scaling parameter s m Rotation parameter r m spherical harmonic coefficient SH m These basic attributes represent the color properties of the scene surface under different viewpoints.

[0105] In the rendering process of 3D Gaussian, given arbitrary camera pose and camera intrinsic parameters, the 3D Gaussian is transformed into a 2D Gaussian on the image plane using Gaussian splashing technology. The intersecting Gaussians of each pixel are recorded, and culling is performed based on the rendering depth of the signed distance field. For each pixel, the RGB values ​​of each intersecting Gaussian are accumulated according to their opacity to obtain the chroma of the current pixel. The opacity of each intersecting Gaussian is accumulated for each pixel to obtain the weight of the current pixel. The color rendering results of the signed distance field and the color rendering results of the 3D Gaussian are weighted and averaged to obtain the global gradient rendering result. The specific calculation steps are as follows:

[0106] (3.1.1) is similar to (1.2.1), given any camera pose T g Given the camera intrinsic parameter matrix K, any pixel u in the image corresponds to a ray r(u) projected from the camera optical center.

[0107] (3.1.2) The color of ray r(u) is calculated by weighted accumulation and depth culling techniques. and weight Complete the rendering of the color image and the weighted image:

[0108]

[0109]

[0110] Where, d iThis represents the depth of the Gaussian plane in the camera coordinate system. This represents the depth obtained from the symbolic distance field rendering; ∈ is a small positive value used to prevent erroneous culling. It is an indicator function. This means removing all depth values ​​greater than 1. Gauss, c i According to SH i The Gaussian color obtained from the observation angle, f i After being projected into pixel space by Gaussian splashing, it is based on the Gaussian center μ i The covariance matrix Σ 2D,i The resulting opacity.

[0111] (3.1.3) Color rendering results of 3D Gaussian Rendering results of the symbolic distance field The data is blended to obtain the rendered global map.

[0112]

[0113] This formula treats the weighting of the symbolic distance field as 1, and the weighting of the three-dimensional Gaussian as... Perform a weighted average.

[0114] 3.2 Determine the regions in the image acquired in the current frame that need to have Gaussian gas added.

[0115] During the scanning process, after updating the symbolic distance field, a global map is rendered and compared with the image acquired in the current frame to determine the regions in the image acquired in the current frame that need to have Gaussian gas added.

[0116]

[0117] Among them, M C This indicates the region where the color of the observation in the current frame is inconsistent with the color of the global map rendering result, δ. C To determine the threshold.

[0118] 3.3 Adding a new 3D Gaussian map to the global map

[0119] To reduce memory overhead, only in M C Twenty-five percent of the pixels are sampled and used to add new Gaussians. The spatial position of each selected pixel is determined by backprojection from the depth map, and the opacity of the Gaussian is initialized to 0.5. The spherical harmonic coefficients of the Gaussian are initialized using a color image, and its shape is initialized as a flat disk conforming to the scene surface. The disk's normal vector corresponds to the pixel's normal vector, and its radius is estimated based on the distance between the newly added Gaussian and other 3D Gaussians. The corresponding scaling parameter s is then calculated based on this. i and rotation parameter ri .

[0120] 3.4 Selecting an Optimized Perspective

[0121] Randomly select N frames from the keyframe sequence, starting from the nearest n frames. local M frames are uniformly selected from a set of ordinary frames. These frames represent the target viewpoints for this optimization. The camera pose and color image corresponding to each viewpoint are obtained and formed into a sequence {T}. g,j C j Specifically, it includes the following sub-steps:

[0122] (3.4.1) Randomly select a camera view obtained in S3.3, and simultaneously render the symbolic distance field and the three-dimensional Gaussian in the global map. The color rendering results of the symbolic distance field and the color rendering results of the three-dimensional Gaussian are weighted and averaged. The weight of the rendering result of the symbolic distance field is 1, and the weight of the color rendering result of the three-dimensional Gaussian is the weight map obtained by rendering.

[0123] (3.4.2) Calculate the L1 distance between the weighted average rendering result and the chroma image corresponding to the selected viewpoint as the rendering error function;

[0124] (3.4.3) Calculate the gradient and update the 3D Gaussian in the global map using the Adam optimizer;

[0125] (3.4.4) Repeat (3.4.1) to (3.4.3) 20 times to complete the iterative optimization.

[0126] 3.5 Establish a rendering error function and optimize the 3D Gaussian rendering of the global map.

[0127] The loss is defined using L1 distance, and the optimization objective is to make the rendered global map as close as possible to the input image. The final loss function is:

[0128]

[0129] Each Gaussian optimization involves P iterations. Each iteration starts from the sequence {T}. g,j C j Randomly select an optimized viewpoint and a reference color image from the map, calculate the loss based on the loss function described above, and use the Adam optimizer to optimize the map and update the 3D Gaussian parameters in the global map.

[0130] 3.6 Remove erroneous or redundant 3D Gaussian vectors

[0131] Identify erroneous Gaussians with excessively large scales and redundant Gaussians with excessively small scales or low opacity in the global map based on the following conditions:

[0132]

[0133] in This represents the length of the longest axis in a three-dimensional Gaussian system.

[0134] The present invention provides a real-time reconstruction system for a 3D scene that fuses a symbolic distance field and a 3D Gaussian representation, comprising one or more processors for implementing the real-time reconstruction method for a 3D scene that fuses a symbolic distance field and a 3D Gaussian representation as described in the above embodiments.

[0135] The embodiments of the real-time reconstruction system for 3D scenes fused with symbolic distance field and 3D Gaussian representation of the present invention can be applied to any device with data processing capabilities, such as a computer. The embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data-processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, in addition to the processor, memory, network interface, and non-volatile memory, the real-time reconstruction system for 3D scenes fused with symbolic distance field and 3D Gaussian representation provided in this embodiment may also include other hardware depending on the actual functions of the data-processing device, which will not be elaborated further.

[0136] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0137] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the real-time reconstruction method for a 3D scene by fusing a symbolic distance field and a 3D Gaussian representation as described in the above embodiments.

[0138] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computing program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0139] Example 1:

[0140] To verify the positioning accuracy, geometric accuracy, temporal performance, spatial performance, and rendering quality of the method of this invention on real and virtual datasets, a desktop computer equipped with an AMD 9950X3D CPU and an NVIDIA RTX 4090 GPU was used to reconstruct a 30-square-meter lounge and an 88-square-meter office. This invention can complete online reconstruction at approximately 150 FPS with 1080x720 resolution image data input, achieving a positioning accuracy of 0.17 cm on the virtual dataset (Replica) and 3.08 cm on the real dataset (TUM RGBD). Figure 3 The reconstruction results of a 33-square-meter activity room show that the present invention has accurate positioning precision and high realism in scene details, while existing methods show obvious reconstruction errors and omissions in details.

[0141] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for real-time reconstruction of a 3D scene by fusing symbolic distance field and 3D Gaussian representation, characterized in that, Includes the following steps: (1) The symbolic distance field of the global map is rendered into depth and color images under the view of each frame captured by the RGBD camera; the depth image of the current frame is matched and registered with the depth image of the global map rendered under the view of the previous frame, and the camera pose is estimated and the key frame sequence is updated. (2) Fuse the color information and depth information of the current frame image into the symbolic distance field of the global map, and update the properties of the visible voxels in the symbolic distance field under the current view. (3) Add a new three-dimensional Gaussian to the three-dimensional Gaussian of the global map, select an optimized view from the historical view and render the global map; the rendering process is a weighted average of the rendering results of the three-dimensional Gaussian of the global map and the rendering results of the symbolic distance field of the global map. (4) Establish and optimize the rendering error function, update the relevant parameters of the 3D Gaussian in the global map; delete redundant Gaussian in the global map; and finally complete the real-time reconstruction of the 3D scene. The error between the rendering result after weighted averaging of the rendering error function and the color image of the current frame.

2. The real-time reconstruction method for a 3D scene by fusing symbolic distance field and 3D Gaussian representation according to claim 1, characterized in that, Step (1) specifically includes the following sub-steps: (1.1) Image data acquisition and preprocessing: During the scanning process of the RGBD camera, the color image and depth image are acquired in the current frame; after the image is transmitted to the computing unit, the depth map is back-projected according to the camera calibration intrinsic parameters to obtain the vertex map and calculate the normal map; the normal vector of each vertex on the vertex map is calculated based on the cross product of the coordinates of the surrounding pixel vertices; (1.2) Image rendering of the symbolic distance field of the global map: The global map maintains a voxel-represented symbolic distance field as one of the basic building blocks of the map; specifically, the vicinity of the scene surface is divided into a series of small cubes with fixed side lengths, called voxels, and a spatial voxel is returned for any three-dimensional spatial point near the surface; each voxel contains the following attributes: color, symbolic distance, and update count, where the absolute value of the symbolic distance represents the distance from the voxel center to the scene surface, a positive value indicates that the voxel is outside the scene surface, and a negative value indicates that the voxel is inside the scene surface; during the rendering of the symbolic distance field, given any camera pose and camera intrinsic parameters, the zero-value surface in the symbolic distance field corresponding to each pixel is obtained through ray casting technology; the three-dimensional coordinates of the zero-value surface are projected onto the camera coordinate system to obtain the depth of the current pixel, and the three-dimensional coordinates of the zero-value surface are interpolated with the colors of nearby voxels to obtain the chroma of the current pixel; (1.3) Camera pose estimation: Based on the model rendering results and the camera intrinsic parameters and camera pose of the previous frame, the 3D coordinates and normal vectors of each vertex in the depth map of the previous frame are calculated using the same calculation method as the current frame to obtain the vertex map and normal map, which form the point cloud under the view of the previous frame; at the same time, the vertex map and normal map of the current frame form the point cloud under the view of the current frame; based on the correspondence between pixels, the correspondence between point clouds is established, and the point-plane iterative nearest point algorithm is used to estimate the camera pose matrix; (1.4) Maintaining the keyframe sequence: The first frame at the start of the scan is fixed as the keyframe. If the spatial distance between the pose of the current frame and the previous keyframe changes by more than 0.3 meters or the angle changes by more than 30 degrees, then this frame is marked as a new keyframe and added to the keyframe sequence.

3. The real-time reconstruction method for a 3D scene by fusing symbolic distance field and 3D Gaussian representation according to claim 1, characterized in that, In step (1.2), the chroma of the current pixel is obtained by interpolating the three-dimensional coordinates of the zero-value surface with the colors of nearby voxels. The calculation specifically includes the following steps: (1.2.1) Given any camera pose T g Given the camera intrinsic parameter matrix K, any pixel u in the image corresponds to a ray r projected from the camera's optical center. u : Among them, R g The rotation matrix representing the camera pose; t g The translation matrix represents the camera pose, and θ is the distance parameter of the light rays; (1.2.2) Proceed in steps along the ray direction with a certain step size δθ until, for a certain ray distance θ, the SDF(r) is satisfied. u (θ))>0 and SDF(r) u (θ+δθ))<0; (1.2.3) Calculate the more accurate surface distance SDF for points on the light ray. + (r u (θ+δθ)) and SDF + (r u (θ)), SDF + The calculation formula is as follows: SDF + (p)=Triliner_Interpolate({SDF(p i )}),i=0,1,…,7; Among them, {SDF(p i )} represents the sign distances between the eight adjacent voxels of p, and Triliner_Interpolate represents trilinear interpolation based on the distance from p to the center of each voxel; (1.2.4) Calculate the zero surface r u (θ * The zero-value surface is then projected onto the camera coordinate system to obtain the rendering depth. (1.2.5) Calculate the zero surface r u (θ * The corresponding color is used as the rendering color. c + (p)=Triliner_Interpolate({c(p i )}),i=0,1,…,7; The calculation method is the same as step (1.2.3).

4. The real-time reconstruction method for a 3D scene by fusing symbolic distance field and 3D Gaussian representation according to claim 1, characterized in that, Step (2) includes the following sub-steps: (2.1) Voxel mesh projection: For all voxels within the visible range of the current frame, project the voxel centers into the image space to obtain pixel coordinates; (2.2) Calculate the symbolic distance of the current frame: For each voxel, calculate the symbolic distance of the current frame based on the depth map of the current frame; (2.3) Weighted update result: With the weight of the new frame as 1, update the symbolic distance, color, and update count attributes within each voxel.

5. The real-time reconstruction method for a 3D scene by fusing symbolic distance field and 3D Gaussian representation according to claim 1, characterized in that, Step (3) specifically includes the following sub-steps: (3.1) The global map also maintains some three-dimensional Gaussians as one of the basic building blocks of the map. Each three-dimensional Gaussian includes attributes such as opacity, spatial position, scaling parameters, rotation parameters, and spherical harmonic coefficients. During the rendering of the three-dimensional Gaussian, given any camera pose and camera intrinsic parameters, the three-dimensional Gaussian is transformed into a two-dimensional Gaussian on the image plane through Gaussian splashing technology. The Gaussians intersecting at each pixel are recorded and culled based on the rendering depth of the signed distance field. For color rendering, the RGB values ​​of each Gaussian intersection point of each pixel are summed according to the opacity to obtain the chroma of the current pixel; For weighted rendering, the weight of the current pixel is obtained by summing the opacities of each Gaussian intersection point for each pixel. The global gradient rendering result is obtained by weighted averaging the color rendering results of the symbolic distance field and the color rendering results of the three-dimensional Gaussian. (3.2) Compare the rendered image of the global map in the current frame with the image acquired in the current frame. Areas with large color errors are areas that need to have Gaussian added. (3.3) Sample pixels with a set threshold in the area where Gaussian needs to be added to add new Gaussian; (3.4) Randomly select a given number of frames from the keyframe sequence and uniformly select a given number of frames from the most recent ordinary frames to obtain the camera viewpoint and corresponding color images of these frames; (3.5) Render the global map under these selected viewpoints, establish a rendering error function and iteratively optimize it; (3.6) Delete the three-dimensional Gaussian in the global map with a scale smaller than 0.003, a scale greater than 0.1, and an opacity less than 0.

005.

6. The real-time reconstruction method for a 3D scene by fusing symbolic distance field and 3D Gaussian representation according to claim 5, characterized in that, In step (3.1), the color rendering results of the symbolic distance field and the color rendering results of the 3D Gaussian are weighted and averaged to obtain the global gradient rendering result; specifically: (3.1.1) Given any camera pose T g Given the camera intrinsic parameter matrix K, any pixel u in the image corresponds to a ray r(u) projected from the camera optical center; (3.1.2) The color of ray r(u) is calculated by weighted accumulation and depth culling techniques. and weight Complete the rendering of the color image and the weighted image: Where, d i This represents the depth of the Gaussian plane in the camera coordinate system. This represents the depth obtained from the symbolic distance field rendering; ∈ is a small positive value used to prevent erroneous culling. It is an indicator function. This means removing all depth values ​​greater than 1. Gauss, c i According to SH i The Gaussian color obtained from the observation angle, f i After being projected into pixel space by Gaussian splashing, it is based on the Gaussian center μ i The covariance matrix Σ 2D,i The resulting opacity; (3.1.3) Color rendering results of 3D Gaussian Rendering results of the symbolic distance field The data is then blended to obtain the rendered global map: The above expression treats the weighting of the symbolic distance field as 1, and the weighting of the three-dimensional Gaussian field as... Perform a weighted average.

7. The real-time reconstruction method for a 3D scene by fusing symbolic distance field and 3D Gaussian representation according to claim 5, characterized in that, Step (3.4) specifically includes the following sub-steps: (3.4.1) Randomly select a camera view obtained in step (3.3), and simultaneously render the symbolic distance field and the three-dimensional Gaussian in the global map. The color rendering results of the symbolic distance field and the color rendering results of the three-dimensional Gaussian are weighted and averaged. The weight of the rendering result of the symbolic distance field is 1, and the weight of the color rendering result of the three-dimensional Gaussian is the weight map it renders. (3.4.2) Calculate the L1 distance between the weighted average rendering result and the chroma image corresponding to the selected viewpoint as the rendering error function; (3.4.3) Calculate the gradient and update the 3D Gaussian in the global map using the Adam optimizer; (3.4.4) Repeat steps (3.4.1) to (3.4.3) 20 times to complete the iterative optimization.

8. An electronic device, characterized in that, It includes one or more processors for implementing the real-time reconstruction method of a three-dimensional scene by fusing a symbolic distance field and a three-dimensional Gaussian representation as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the real-time reconstruction method for three-dimensional scenes by fusing symbolic distance field and three-dimensional Gaussian representation as described in any one of claims 1 to 6.

Images