Laser enhanced vision three-dimensional reconstruction method and system based on Gaussian splashing

Abstract

The invention provides a laser enhanced vision three-dimensional reconstruction method and system based on Gaussian splashing, and belongs to the field of computer vision. The method is divided into two parts of pose calculation and scene reconstruction: during pose calculation, relative rotation is solved by using image feature matching, and the relative pose with a real scale is further solved according to the pose of the relative rotation initialization point cloud ICP; the pose is further optimized by combining the re-projection error of the image matching feature and the matching error of the point cloud through a least square method, so that the high-precision pose is obtained; during scene reconstruction, a large number of three-dimensional Gaussian balls are used for representing a scene, an image with a high-precision pose and a depth map generated by point cloud projection are used for supervising a composite image capable of being micro-rendered, and a gradient descent method is used for optimizing parameters of the three-dimensional Gaussian balls to reconstruct and optimize the three-dimensional scene. Therefore, dense reconstruction of the scene and high-quality new view angle image synthesis are realized.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to a laser-enhanced vision three-dimensional reconstruction method and system based on Gaussian splashing. Background Art

[0002] 3D reconstruction uses various sensor devices to capture information about the real world and build a 3D model based on this information. 3D reconstruction has broad application prospects in various fields. Traditional 3D reconstruction algorithms are mainly based on vision and lidar. 3D reconstruction algorithms based on a single sensor each have their own unique advantages and limitations. Vision-based algorithms use image information obtained by the camera to infer the depth and 3D structure of objects through computer vision technology. However, visual sensors cannot effectively deal with situations where textures are repeated or missing. This means that in areas where there are a large number of similar textures, visual sensors may not be able to accurately capture the details of the scene, resulting in inaccurate reconstruction results. Lidar-based algorithms use sensors such as lidar to directly obtain the distance information from the object to the sensor, so they can provide accurate depth information and are not affected by lighting and texture, but they cannot capture the color information of the scene. Therefore, when using laser sensors alone for scene reconstruction, scene models with real colors cannot be generated, which may affect the visualization of the results in some application scenarios. In practical applications, multiple sensors and algorithms are often combined to achieve more accurate and robust 3D reconstruction to meet different scenarios and application requirements. Traditional algorithms that integrate vision and laser face some challenges in achieving photo-level scene reconstruction. First, the fusion of visual data and laser data is challenging. Due to the large differences in their information representation and features, the fusion process is affected by problems such as data inconsistency and calibration errors. Second, traditional algorithms may encounter difficulties when dealing with complex scenes, such as dealing with occlusion, reflection, and transparent objects, which may lead to ambiguity in the algorithm's understanding of the scene. In addition, traditional algorithms usually require a lot of computing resources and time to process large amounts of data, and may lack sufficient accuracy and robustness in the face of noise and illumination changes. In summary, the difficulties and challenges faced by traditional 3D reconstruction algorithms are as follows: (1) 3D reconstruction algorithms based on a single sensor are difficult to cope with special scenes. Visual sensors cannot effectively handle situations where textures are repeated or missing, and lidar cannot capture the color information of the scene. (2) 3D reconstruction algorithms that integrate vision and laser face the challenge of cross-modal data fusion. The information representation and features of images and laser point clouds are very different, resulting in the fusion process being affected by problems such as data inconsistency and calibration errors. (3) Traditional 3D reconstruction algorithms are difficult to reconstruct photo-quality dense 3D scenes. Traditional 3D scene representation methods focus on the representation of the geometric structure of the scene, which makes it difficult to achieve photo-quality dense reconstruction. Summary of the invention

[0003] In view of the difficulties in cross-modal data fusion and high-quality new perspective rendering in existing 3D reconstruction algorithms, the present invention provides a laser-enhanced visual 3D reconstruction method based on Gaussian splashing. High-precision positioning and high-fidelity mapping by integrating visual and laser data are achieved. The relative pose is optimized by combining visual and laser data to improve the accuracy of pose calculation; a 3D Gaussian sphere is used to represent the scene, and a 3D Gaussian splash is used to render an image from the scene. The input image is used as supervision, and the parameters of the Gaussian sphere in the scene are optimized using a gradient descent algorithm to achieve high-fidelity 3D scene reconstruction.

[0004] The present invention provides a laser enhanced visual three-dimensional reconstruction method based on Gaussian splashing, comprising:

[0005] Step 1: extract features from all input images, calculate feature matching between adjacent images, and use epipolar constraints to solve the relative rotation transformation between adjacent images;

[0006] Step 2: Initialize the pose between adjacent point clouds based on the relative rotation transformation obtained in step 1 as the initial value, and use the point-to-plane ICP algorithm to solve the relative pose between the point clouds;

[0007] Step 3: Based on the obtained feature matching and the matching relationship between the points in the point clouds obtained by the nearest neighbor search, an objective function including the reprojection error of the matching features and the point cloud registration error is constructed, and the relative pose is optimized using the nonlinear least squares method.

[0008] Step 4: Based on the relative pose optimized in step 3, the relative pose is adjusted using the Gaussian splash rendering result, the camera and the scene are aligned, and the pose is further optimized;

[0009] Step 5, based on the pose calculated in step 4, render a contour map of the scene from the three-dimensional Gaussian scene, and add an initial Gaussian sphere to the unrepresented part of the scene according to the point cloud and the image;

[0010] Step 6: For the newly added Gaussian ball, optimize the properties of the Gaussian ball according to the image and depth, select several key frames, fix the camera pose, use the depth map and RGB image as supervision, and use the gradient descent algorithm to optimize the color c, position μ, and opacity o of the Gaussian ball in the three-dimensional Gaussian scene, thereby achieving high-fidelity three-dimensional scene reconstruction.

[0011] Furthermore, the specific implementation of step 1 is as follows;

[0012] The camera intrinsic parameters are used to dedistort the input original image. Then, SIFT feature points are extracted from the dedistorted image and the descriptor is calculated. Feature matching is calculated for adjacent images, and the RANSAC method is used to remove external points to obtain more accurate feature matching results. The RANSAC algorithm is then used to calculate the epipolar constrained essential matrix based on the retained feature matching, and the SVD algorithm is used to decompose the relative rotation transformation from the essential matrix.

[0013] Furthermore, in step 2, the rotation matrix of the initial pose of the point-to-plane ICP algorithm is the result solved in step 1, the translation vector is zero, and the relative pose between adjacent frames is solved using the ICP algorithm.

[0014] Furthermore, the specific implementation of step 3 is as follows;

[0015] For adjacent images and point clouds, the 3D coordinates of the feature points are solved according to the relative pose and feature matching. The 3D feature points are projected onto two camera planes according to the internal and external parameters of the camera, and the distance between the projected point coordinates and the feature coordinates is used as the reprojection error:

[0016] e c =π(x; T k ,P)-u

[0017] Where x represents a 3D feature point, T k represents the pose of the kth frame, P represents the projection matrix, π represents the projection transformation, and u is the coordinate of the corresponding image pixel;

[0018] After the point cloud is transformed in relative position, the nearest neighbor point is used as the matching point, and the distance between the matching points is calculated as the registration error of the point cloud:

[0019]

[0020] Where (p, q) is a pair of 3D feature points matched in the point cloud of k, k-1 frames, T k , T k-1 They are the poses of the point clouds of frames k and k-1 respectively, τ is the transformation function, and p is based on Transform to new coordinates, T e It is the external reference between radar and camera;

[0021] The final objective function is composed of the two errors mentioned above, and the joint optimization objective function is as follows:

[0022]

[0023] Among them, λ c and λ l is the weight, C and L are the labels of the image reprojection error and the laser point cloud registration error respectively;

[0024] Use the relative pose in step 2 as the initial value for optimization and use nonlinear least squares optimization to obtain the final pose.

[0025] Furthermore, the specific implementation of step 4 is as follows;

[0026] According to the relative pose calculated in step 3, the scene outline, depth map and RGB image are rendered from the three-dimensional Gaussian scene. A three-dimensional scene G is represented by a set of Gaussian balls. Each Gaussian ball in the scene contains the attributes color c, position μ, radius r and opacity o∈[0,1]. When rendering, the Gaussian balls are sorted from small to large according to the depth to the camera. For a pixel p on the image, the color is calculated by the following formula:

[0027]

[0028] Where C represents the rendered RGB image, n is the number of Gaussian balls, and c i represents the color of the i-th Gaussian ball, f i Indicates the influence of the i-th Gaussian ball on pixel p. The function f is defined as follows:

[0029]

[0030] in, d=(Tμ) z ; P is the projection matrix, T is the given pose, μ is the position of the Gaussian sphere, f is the camera focal length, o is the opacity of the Gaussian sphere, and x is the independent variable;

[0031] Replace the color feature in the formula of rendering color with the depth z or unit 1, and render the depth map D and the contour map S;

[0032] Use the contour map S as a mask and the depth map of the point cloud projection And the collected RGB image As the true value, the camera pose is used as the parameter to be optimized, and the gradient descent algorithm is used to adjust the camera pose to align the camera with the scene:

[0033]

[0034] Among them, T k represents the pose of the kth frame, C is the rendered color map, and D is the rendered depth map.

[0035] Finally, the input frame containing the RGB image and the depth map is denoted as F, F = {C, D}, and the rendering function is denoted as Render(G, T), then the goal of optimizing the pose can be simplified as:

[0036]

[0037] Among them, Render represents the rendering process.

[0038] Furthermore, the rendering formulas of the depth map D and the contour map S are as follows:

[0039]

[0040] where z i Represents the depth of the i-th Gaussian sphere.

[0041] Furthermore, the specific implementation of step 5 is as follows;

[0042] First, use the pose calculated in step 4 to render the scene's outline. Add and initialize a Gaussian sphere to the unrepresented part of the scene. Use the points in the point cloud to initialize the pose of the Gaussian sphere and the RGB image to initialize the color of the Gaussian sphere. The unrepresented part is given by the following formula:

[0043] M(p)=(S(p)<λ)

[0044] When the value of a pixel in the rendered contour image is less than λ, the space corresponding to the pixel is an unrepresented part.

[0045] Then, the process of merging to obtain a new Gaussian scene is the process of Gaussian densification, that is, adding the initial Gaussian ball, which is recorded as:

[0046]

[0047] Among them, Densify means Gaussian densification, Represents the result of Gaussian densification.

[0048] Furthermore, in step 6, a key frame is selected every n frames, and when updating the Gaussian scene, k frames are selected to participate in the optimization, including the current frame, the most recent key frame, and the k-2 frames with the largest overlap rate with the current frame, where the overlap rate is calculated by projecting the point cloud of the current frame onto other frames.

[0049] The present invention also provides a laser enhanced visual three-dimensional reconstruction system based on Gaussian splashing, comprising:

[0050] one or more processors;

[0051] A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a laser-enhanced visual three-dimensional reconstruction method based on Gaussian splashing as described in the above scheme.

[0052] The principle of the present invention is: when performing three-dimensional reconstruction, the camera's posture must first be obtained, and then the three-dimensional scene must be reconstructed based on the posture. LiDAR enhanced visual three-dimensional reconstruction can calculate high-precision relative posture by fusing information from laser point clouds and visual images. The three-dimensional Gaussian splashing scene representation method uses a large number of Gaussian balls to represent the three-dimensional scene. Through three-dimensional Gaussian splashing, RGB images and depth maps of the scene can be rendered from the scene. Using the three-dimensional Gaussian splashing scene representation method, photo-level reconstruction of the scene can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is an overall structural block diagram of an embodiment of the present invention;

[0054] Figure 2 The rendering result of the reconstructed scene according to the embodiment of the present invention;

[0055] Figure 3 This is the reconstructed scene and posture trajectory of an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention provides a laser enhanced visual 3D reconstruction algorithm based on Gaussian splashing. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings.

[0057] New perspective synthesis technology can synthesize realistic photos from new perspectives. It uses implicit representations (such as neural radiation fields) or explicit representations (such as grids, signed distance functions, and three-dimensional Gaussian point clouds) to represent three-dimensional scenes. These methods reconstruct the geometric structure of the scene and maintain visual integrity to synthesize photo-realistic images. However, in the field of new perspective synthesis, the pursuit of high peak signal-to-noise ratio often ignores the map structure. By introducing this technology into the three-dimensional reconstruction process, photo-level three-dimensional reconstruction that is difficult to achieve with traditional three-dimensional reconstruction algorithms can be achieved. The overall structural block diagram of the laser enhanced visual three-dimensional reconstruction method based on Gaussian splashing proposed in an embodiment of the present invention is shown in the attached figure. Figure 1 As shown, the specific steps include:

[0058] Step 1: Extract features from all input images, calculate feature matching between adjacent images, and use epipolar constraints to solve the relative rotation transformation between adjacent images.

[0059] The specific approach is: use the camera intrinsic parameters to dedistort the input original image i. Then extract SIFT feature points from the dedistorted image and calculate the descriptor. Calculate feature matching for adjacent images, and use the RANSAC method to remove external points to obtain more accurate feature matching results. Then use the RANSAC algorithm to calculate the epipolar constraint essential matrix based on the retained feature matching, and use the SVD algorithm to decompose the relative rotation matrix from the essential matrix.

[0060] Step 2: Use the relative rotation obtained in the above step as the initial value to initialize the pose between adjacent point clouds, and use the point-to-surface ICP algorithm to solve the relative pose between point clouds.

[0061] The specific method is as follows: the rotation matrix of the initial pose of the point-to-plane ICP algorithm is the result of step 1, and the translation vector is zero. The relative pose between adjacent frames is solved using the ICP algorithm.

[0062] Step 3: Based on the feature matching obtained in the above steps and the matching relationship between the points in the point clouds obtained by the nearest neighbor search, construct an objective function that includes the reprojection error of the matching features and the point cloud registration error, and use the nonlinear least squares method to optimize the relative pose.

[0063] The specific approach is: for adjacent images and point clouds, the 3D coordinates of the feature points are solved according to the relative pose and feature matching, and the 3D feature points are projected onto two camera planes according to the internal and external parameters of the camera, and the distance between the projected point coordinates and the feature coordinates is used as the reprojection error:

[0064] e c =π(x; T k ,P)-u

[0065] Where x represents a 3D feature point, T k represents the pose of the kth frame, P represents the projection matrix, π represents the projection transformation, and u is the coordinate of the corresponding image pixel;

[0066] After the point cloud is transformed in relative position, the nearest neighbor point is used as the matching point, and the distance between the matching points is calculated as the registration error of the point cloud:

[0067]

[0068] Where (p, q) is a pair of 3D feature points matched in the point cloud of k, k-1 frames, T k , T k-1 They are the poses of the point clouds of frames k and k-1 respectively, τ is the transformation function, and p is based on Transform to new coordinates, T e It is the external parameter between the radar and the camera, which is calibrated in advance;

[0069] The final objective function is composed of the two errors mentioned above, and the joint optimization objective function is as follows:

[0070]

[0071] Among them, λ c and λ lis the weight, C and L are the labels of the image reprojection error and the laser point cloud registration error respectively;

[0072] Use the relative pose in step 2 as the initial value for optimization and use nonlinear least squares optimization to obtain the final pose.

[0073] Step 4: Based on the relative pose calculated in step 3, use the Gaussian splash rendering result to adjust the relative pose, align the camera and the scene, and further optimize the pose.

[0074] The specific method is: according to the relative pose calculated in step 3, the scene outline, depth map and RGB image are rendered from the 3D Gaussian scene. A 3D scene G is represented by a set of Gaussian balls. Each Gaussian ball in the scene contains the attributes color c, position μ, radius r and opacity o∈[0,1]. When rendering, the Gaussian balls are sorted from small to large according to the depth to the camera. For a pixel p on the image, the color is given by the following formula:

[0075]

[0076] Where C represents the rendered RGB image, n is the number of Gaussian balls, and c i represents the color of the i-th Gaussian ball, f i Indicates the influence of the i-th Gaussian ball on pixel p. The function f is defined as follows:

[0077]

[0078] in, d=(Tμ) z ; P is the projection matrix, T is the given pose, μ is the position of the Gaussian sphere, f is the camera focal length, o is the opacity of the Gaussian sphere, and x is the independent variable;

[0079] Replace the color feature in the formula for rendering color with the depth z or unit 1, and render the depth map D and the contour map S:

[0080]

[0081] where z i represents the depth of the i-th Gaussian sphere;

[0082]

[0083] Use the contour map S as a mask and the depth map of the point cloud projection And the collected RGB image As the true value, the camera pose is used as the parameter to be optimized, and the gradient descent algorithm is used to adjust the camera pose to align the camera with the scene:

[0084]

[0085] T k represents the pose of the kth frame, C is the rendered color map, and D is the rendered depth map.

[0086] Finally, the input frame containing the RGB image and the depth map is denoted as F, F = {C, D}, and the rendering function is denoted as Render(G, T), then the goal of optimizing the pose can be simplified as:

[0087]

[0088] Among them, Render represents the rendering process.

[0089] Step 5: Based on the pose calculated in step 4, render a contour map of the scene from the 3D Gaussian scene, and add an initial Gaussian sphere to the unrepresented parts of the scene based on the point cloud and image.

[0090] The specific approach is: use the pose calculated in the above steps to render the outline of the scene. The calculation method is the same as step 4. Add and initialize the Gaussian sphere to the unrepresented part of the scene. Use the points of the point cloud to initialize the pose of the Gaussian sphere, and the RGB image to initialize the color of the Gaussian sphere. The unrepresented part is given by the following formula:

[0091] M(p)=(S(p)<0.5)

[0092] When the value of a pixel in the rendered contour image is less than 0.5, the space corresponding to the pixel is an unrepresented part.

[0093] Then, the process of merging to obtain a new Gaussian scene is the process of Gaussian densification, that is, adding the initial Gaussian ball, which is recorded as:

[0094]

[0095] Among them, Densify means Gaussian densification, Represents the result of Gaussian densification.

[0096] Step 6: For the newly added Gaussian ball, it is necessary to optimize the properties of the Gaussian ball according to the image and depth. Select several key frames, fix the camera pose, use the depth map and RGB image as supervision, and use the gradient descent algorithm to optimize the color c, position μ, and opacity o of the Gaussian ball in the three-dimensional Gaussian scene.

[0097] The specific method is: a key frame is selected every 4 frames, and when updating the Gaussian scene, 24 frames are selected for optimization, including the current frame, the most recent key frame, and the 22 frames with the largest overlap rate with the current frame. The overlap rate is calculated by projecting the point cloud of the current station onto other frames.

[0098] The present invention has conducted multiple experiments on the Replica dataset, and the reconstructed rendering results are shown in the attached figure. Figure 2 As shown in the figure, the three-dimensional scene and pose trajectory are shown in the attached Figure 3 As shown in the figure, in terms of quantitative indicators, the experiment measured the rendering quality and geometric structure of the reconstructed image, and the pose trajectory error and the rendering depth error were less than 3 cm, which well proved the effectiveness of the 3D reconstruction method of the present invention.

[0099] Another invention, an embodiment of the present invention also provides a laser enhanced visual 3D reconstruction system based on Gaussian splashing, comprising:

[0100] one or more processors;

[0101] A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a laser-enhanced visual three-dimensional reconstruction method based on Gaussian splashing as described in the above scheme.

[0102] The above contents are further detailed descriptions of the present invention in combination with the best implementation scheme, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. It should be understood by those skilled in the art that various modifications can be made to the details without departing from the scope of the appended claims, and all should be deemed to belong to the protection scope of the present invention.

Claims

1. A laser-enhanced visual 3D reconstruction method based on Gaussian splashing, characterized in that: The following steps are involved: Step 1: extract features from all input images, calculate feature matching between adjacent images, and use epipolar constraints to solve the relative rotation transformation between adjacent images; Step 2: Initialize the pose between adjacent point clouds based on the relative rotation transformation obtained in step 1 as the initial value, and use the point-to-plane ICP algorithm to solve the relative pose between the point clouds; Step 3: Based on the obtained feature matching and the matching relationship between the points in the point clouds obtained by the nearest neighbor search, an objective function including the reprojection error of the matching features and the point cloud registration error is constructed, and the relative pose is optimized using the nonlinear least squares method. Step 4: Based on the relative pose optimized in step 3, the relative pose is adjusted using the Gaussian splash rendering result, the camera and the scene are aligned, and the pose is further optimized; Step 5, based on the pose calculated in step 4, render a contour map of the scene from the three-dimensional Gaussian scene, and add an initial Gaussian sphere to the unrepresented part of the scene according to the point cloud and the image; Step 6: For the newly added Gaussian ball, optimize the properties of the Gaussian ball according to the image and depth, select several key frames, fix the camera pose, use the depth map and RGB image as supervision, and use the gradient descent algorithm to optimize the color c, position μ, and opacity o of the Gaussian ball in the three-dimensional Gaussian scene, thereby achieving high-fidelity three-dimensional scene reconstruction.

2. The laser-enhanced visual 3D reconstruction method based on Gaussian splashing according to claim 1, characterized in that: The specific implementation of step 1 is as follows; The camera intrinsic parameters are used to dedistort the input original image. Then, SIFT feature points are extracted from the dedistorted image and the descriptor is calculated. Feature matching is calculated for adjacent images, and the RANSAC method is used to remove external points to obtain more accurate feature matching results. The RANSAC algorithm is then used to calculate the epipolar constrained essential matrix based on the retained feature matching, and the SVD algorithm is used to decompose the relative rotation transformation from the essential matrix.

3. A laser enhanced visual 3D reconstruction method based on Gaussian splashing according to claim 1, characterized in that: In step 2, the rotation matrix of the initial pose of the point-to-surface ICP algorithm is the result of step 1, the translation vector is zero, and the relative pose between adjacent frames is solved using the ICP algorithm.

4. A laser enhanced visual 3D reconstruction method based on Gaussian splashing according to claim 1, characterized in that: The specific implementation of step 3 is as follows; For adjacent images and point clouds, the 3D coordinates of the feature points are solved according to the relative pose and feature matching. The 3D feature points are projected onto two camera planes according to the internal and external parameters of the camera, and the distance between the projected point coordinates and the feature coordinates is used as the reprojection error: and c =π(x):T k ,P)-u Where x represents a 3D feature point, T k represents the pose of the kth frame, P represents the projection matrix, π represents the projection transformation, and u is the coordinate of the corresponding image pixel; After the point cloud is transformed in relative position, the nearest neighbor point is used as the matching point, and the distance between the matching points is calculated as the registration error of the point cloud: Where (p, q) is a pair of 3D feature points matched in the point cloud of k, k-1 frames, T k , T k-1 They are the poses of the point clouds of frames k and k-1 respectively, τ is the transformation function, and p is based on Transform to new coordinates, T e It is the external reference between radar and camera; The final objective function is composed of the two errors mentioned above, and the joint optimization objective function is as follows: Among them, λ c and λ l is the weight, C and L are the labels of the image reprojection error and the laser point cloud registration error respectively; Use the relative pose in step 2 as the initial value for optimization and use nonlinear least squares optimization to obtain the final pose.

5. A laser enhanced visual 3D reconstruction method based on Gaussian splashing according to claim 1, characterized in that: The specific implementation of step 4 is as follows; According to the relative pose calculated in step 3, the scene outline, depth map and RGB image are rendered from the three-dimensional Gaussian scene. A three-dimensional scene G is represented by a set of Gaussian balls. Each Gaussian ball in the scene contains the attributes color c, position μ, radius r and opacity o∈[0,1]. When rendering, the Gaussian balls are sorted from small to large according to the depth to the camera. For a pixel p on the image, the color is calculated by the following formula: Where C represents the rendered RGB image, n is the number of Gaussian balls, and c i represents the color of the i-th Gaussian ball, f i Indicates the influence of the i-th Gaussian ball on pixel p. The function f is defined as follows: in, d=(Tμ) z ; P is the projection matrix, T is the given pose, μ is the position of the Gaussian sphere, f is the camera focal length, o is the opacity of the Gaussian sphere, and x is the independent variable; Replace the color feature in the formula of rendering color with the depth z or unit 1, and render the depth map D and the contour map S; Use the contour map S as a mask and the depth map of the point cloud projection And the collected RGB image As the true value, the camera pose is used as the parameter to be optimized, and the gradient descent algorithm is used to adjust the camera pose to align the camera with the scene: Among them, T k represents the pose of the kth frame, C is the rendered color map, and D is the rendered depth map; Finally, the input frame containing the RGB image and the depth map is recorded as F, F = {C, D}, and the rendering function is recorded as Render(G, T). Then the goal of optimizing the pose can be simplified as: Among them, Render represents the rendering process.

6. A laser enhanced visual 3D reconstruction method based on Gaussian splashing according to claim 5, characterized in that: The rendering formulas of the depth map D and the contour map S are as follows: where z i Represents the depth of the i-th Gaussian sphere.

7. A laser enhanced visual 3D reconstruction method based on Gaussian splashing according to claim 6, characterized in that: The specific implementation of step 5 is as follows; First, use the pose calculated in step 4 to render the scene's outline. Add and initialize a Gaussian sphere to the unrepresented part of the scene. Use the points in the point cloud to initialize the pose of the Gaussian sphere and the RGB image to initialize the color of the Gaussian sphere. The unrepresented part is given by the following formula: M(p)=(S(p)<λ) When the value of a pixel in the rendered contour is less than λ, the space corresponding to the pixel is an unrepresented part; Then, the process of merging to obtain a new Gaussian scene is the process of Gaussian densification, that is, adding the initial Gaussian ball, which is recorded as: Among them, Densify means Gaussian densification, Represents the result of Gaussian densification.

8. A laser enhanced visual 3D reconstruction method based on Gaussian splashing according to claim 1, characterized in that: In step 6, a key frame is selected every n frames. When updating the Gaussian scene, k frames are selected to participate in the optimization, including the current frame, the most recent key frame, and the k-2 frames with the largest overlap rate with the current frame, where the overlap rate is calculated by projecting the point cloud of the current frame onto other frames.

9. A laser-enhanced visual 3D reconstruction system based on Gaussian splashing, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 8.

Images