Complex scene three-dimensional reconstruction method based on multi-source fusion

By combining the multi-source fusion method of neural radiation field and 3D Gaussian sputtering in complex scenes, the multi-modal data association problem and inaccurate position in three-dimensional reconstruction are solved, and high-precision 3D reconstruction and new perspective rendering are achieved result.

CN119006700AActive Publication Date: 2024-11-22INST OF AUTOMATION CHINESE ACAD OF SCI (LUOYANG) ROBOTICS & INTELLIGENT EQUIP INNOVATION INST

Patent Information

Application Number
CN202410921977.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-22
Estimated Expiration
2044-07-10

AI Technical Summary

Technical Problem

Existing three-dimensional reconstruction techniques have multiple challenges in complex scenarios, including scale differences in visual methods, field of view limitations and sparse point cloud distribution of lidars, resulting in traditional fusion methods not being able to make full use of multimodal data, and neural radiation fields and 3D Gaussian sputtering performance deteriorates when the position is inaccurate, resulting in a decrease in accuracy.

Method used

Using a multi-source fusion method based on neural radiation field and 3D Gaussian sputtering, the lidar point cloud and camera image data are encoded in the symbol distance field, multimodal data is correlated through loss function, and 3D GS is used to guide NeRF training. Optimize the three-dimensional Gaussian ellipsoid and pose, and dynamically adjust the Gaussian ellipsoid to improve reconstruction accuracy.

Benefits of technology

It realizes high-precision three-dimensional reconstruction in complex scenarios, jointly optimizes the camera and lidar poses, and directly extracts the three-dimensional grid model, improving the reconstruction accuracy and application value.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006700A_ABST
    Figure CN119006700A_ABST
Patent Text Reader

Abstract

The invention discloses a complex scene three-dimensional reconstruction method based on multi-source fusion, which is characterized in that information fusion of multiple sensors is realized based on a neural radiation field and three-dimensional Gaussian sputtering, the whole scene is coded in a symbol distance field of NeRF, and then SDF is used as an intermediate medium to associate multi-modal data; in the NeRF training process, the 3D GS is used as guidance to explicitly supervise the estimated value of the SDF and guide the rays of the NeRF to sample. According to the method, camera and laser radar poses and implicit expression of the scene can be jointly optimized under the condition that the initial pose is given, a more accurate pose and scene three-dimensional reconstruction result is obtained, and higher reconstruction precision is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method for complex scenes based on multi-source fusion. Background Art

[0002] Dense 3D reconstruction of scenes has been a hot research area, and two main types of sensors are currently used, namely cameras and lidar.

[0003] In the past few decades, cameras have become the most commonly used sensors for 3D reconstruction due to their cost-effectiveness, high resolution, and ability to capture rich semantic information. In traditional vision-based 3D reconstruction, Structure-from-Motion (SfM) or Simultaneous Localization and Mapping (SLAM) methods are usually used to calculate the camera pose and perform sparse reconstruction. Subsequently, Multi-View Stereo (MVS) methods are used for dense reconstruction. However, due to the limitations of camera sensors, pure vision methods have the following difficulties: First, the lack of depth information leads to scale differences between the model reconstructed from the image and the actual model and camera motion; second, vision-based methods require sufficient overlapping areas between images to accurately recover camera motion, which is a challenge for cameras with small fields of view; finally, the reconstructed environment needs to have rich textures and numerous visual features so that reliable feature matching can be achieved between images. The above factors make it challenging to use vision-based 3D reconstruction in some complex scenes.

[0004] LiDAR can capture accurate depth information by emitting laser beams and calculating the laser flight time, so it is almost unaffected by lighting or environment. The LiDAR Odometry and Mapping (LOAM) method, as well as a series of improved algorithms such as LeGO-LOAM and F-LOAM, are currently the most widely used methods for 3D reconstruction based on LiDAR. Since LiDAR can detect objects in three-dimensional space, the LOAM method solves the scale singularity problem. In addition, the LOAM method uses spatial structure as a feature, so 3D reconstruction can still be performed in textureless scenes. However, this method performs poorly in some environments that lack obvious structural features, such as long corridors or playgrounds; in addition, LiDAR points are sparsely and unevenly distributed in space, that is, the farther away from the radar, the fewer spatial points, which makes it difficult to judge the similarity between two LiDAR point clouds. Therefore, the LOAM method has limitations in closed-loop detection and is easily affected by cumulative errors.

[0005] Therefore, combining lidar and camera for 3D reconstruction can make up for their respective shortcomings and obtain better reconstruction results. Most of the existing fusion-based reconstruction methods rely heavily on the strict synchronization of multimodal data, and some of them also require additional auxiliary sensors, such as inertial measurement units (IMUs) and global navigation satellite systems (GNSS). In addition, the fusion of traditional cameras and lidar is often not sufficient. For example, some methods maintain a visual map and a lidar map at the same time. When the input information is an image, the visual map is used for pose solution; when the input information is a lidar, the lidar map is used for solution. This method increases the system overhead, lacks information association between sensors, and fails to fully integrate information from different sensors.

[0006] The fundamental reason why the information from the above different sensors is not fully integrated is that it is very difficult to correlate multimodal data. For images and lidar, the difficulty in correlating the two lies in the following three aspects:

[0007] (1) Different data modalities: images are essentially two-dimensional information, while lidar is three-dimensional information;

[0008] (2) The amount of data is different; each image has about several million pixels, while LiDAR has only about 100,000 3D points;

[0009] (3) The sensor field of view is different; the camera field of view is generally 60 to 100 degrees, while the lidar has a 360-degree field of view in the horizontal direction.

[0010] The above difficulties lead to the traditional fusion method being in a state of "fighting alone", and it is impossible to obtain the best 3D reconstruction results. Moreover, the 3D reconstruction results obtained by the fusion method exist in the form of point clouds, that is, the entire reconstructed model is a discrete 3D point in space. In many applications, because the discrete point cloud model cannot be infinitely subdivided, it is often converted into a triangular mesh model (Mesh) first. In this conversion process, errors are inevitable.

[0011] The introduction of Neural Radiance Field (NeRF) technology has brought a strong driving force for the research of 3D scene reconstruction and new perspective synthesis. The core of NeRF is to use an MLP (multi-layer perceptron) to express the entire scene. Unlike traditional explicit scene expression, NeRF is an implicit expression of a scene, that is, the entire scene is stored in the neural network; the color, transparency and other information of each point in the scene can be obtained by inputting the query point into the neural network. However, since NeRF relies on the photometric consistency of the rendered image and the real image as a supervisory signal during the training process, the scene expression obtained by training is more inclined to "photometric consistency" rather than "geometric correctness"; in some cases, incorrect geometry can also obtain a correct rendered image; therefore, using the original NeRF for 3D reconstruction often fails to obtain the correct result. In addition, NeRF requires the network to be initialized, which is only applicable to the case where the object is located in the center of the scene, such as image capture and reconstruction around a single object; in large-scale scenes, the above assumptions will be destroyed, which will lead to the non-convergence of neural network training.

[0012] 3D Gaussian Splatting (3D GS) technology is another method for new perspective synthesis. This method is a more explicit scene expression. Its core is to use a set of three-dimensional ellipsoids to express the entire three-dimensional scene. Each Gaussian ellipsoid has properties such as mean, variance, color, and transparency. By projecting these ellipsoids onto the camera plane, a rendered image can be obtained. Compared with NeRF, 3D GS runs faster and has higher accuracy in perspective synthesis. In terms of three-dimensional reconstruction, since 3D GS is a more explicit expression, each ellipsoid has a center point, and the point cloud of the scene can be obtained by extracting the center point of the ellipsoid. However, since 3D GS itself focuses on rendering, this will cause the center of each Gaussian ellipsoid to not be located on the surface of the scene, but to be distributed near the surface; this results in the results of three-dimensional reconstruction using the original 3D GS often having low accuracy.

[0013] Moreover, both neural radiation field and 3D Gaussian sputtering require the use of other methods (such as SfM) to obtain the camera pose in advance. In the case of inaccurate pose, the performance of the above two methods will be significantly reduced, which will lead to reduced accuracy when using them for 3D reconstruction. Summary of the invention

[0014] To solve the above problems, the purpose of the present invention is to provide a three-dimensional reconstruction method for complex scenes based on multi-source fusion, which realizes multi-sensor information fusion based on neural radiance field (NeRF) and three-dimensional Gaussian sputtering (3D GaussianSplatting, 3D GS); the entire scene is encoded in the signed distance function (SDF) of NeRF, and then the multimodal data is associated through SDF as an intermediate medium; in the training process of NeRF, 3D GS is used as a guide to explicitly supervise the estimated value of SDF and guide the NeRF rays to sample.

[0015] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0016] A complex scene 3D reconstruction method based on multi-source fusion, comprising the following steps:

[0017] S1. Collecting laser radar point cloud data and camera image data of the target scene, using the laser radar point cloud data to perform pose estimation to obtain laser radar pose information; using the camera image data to perform pose estimation to obtain camera pose information;

[0018] S2. Based on the camera image data and camera pose information in step S1, a loss function related to the three-dimensional ellipsoid is calculated, where the loss function is used to train the properties of the three-dimensional ellipsoid and optimize the camera pose. The properties of the three-dimensional ellipsoid include: color, position, transparency, rotation, and scale. The rotation and scale are combined to obtain the variance of the three-dimensional Gaussian ellipsoid.

[0019] S3, based on the camera image data and the lidar point cloud data in step S1, as well as the camera and lidar poses, calculating a loss function related to the neural radiation field;

[0020] S4, adding all the losses in step S2 and step S3 together to obtain the final loss; back-propagating the final loss to optimize the three-dimensional Gaussian ellipsoid, the neural radiation field, and the camera and lidar poses;

[0021] S5. Dynamically increase the Gaussian ellipsoid according to the gradient of the final loss propagated to the mean of the Gaussian ellipsoid;

[0022] S6. If the transparency of a Gaussian ellipsoid is lower than 0.001, directly delete the Gaussian ellipsoid;

[0023] S7, repeating the above steps S2 to S6 at least tens of thousands of times, completing the training, and obtaining a three-dimensional expression of the entire scene;

[0024] S8. Extract the three-dimensional reconstruction result of the scene based on the trained neural radiation field.

[0025] Furthermore, the above step S2 includes the following sub-steps:

[0026] S2.1. Randomly select an image, and project N Gaussian ellipsoids of the target scene to the image coordinates according to the position and posture of the image. The projection process includes projecting the mean and variance of the Gaussian ellipsoids respectively, obtaining the coordinates of the Gaussian mean under the image plane and the projection of the variance on the two-dimensional plane, and determining the projected Gaussian data;

[0027] S2.2. Based on the projected Gaussian data, a new image is rendered. Each pixel p on the image i The rendered color is the weighted average of the colors of M Gaussian ellipsoids, and N ≥ M;

[0028] S2.3. Obtaining a rendered image After that, the rendered image and the real image are calculated Rendering loss

[0029]

[0030] Among them, SSIM(a,b) represents the structural similarity of two images; λ ssim is a hyperparameter used to balance the two errors;

[0031] S2.4. Assuming there are N Gaussian ellipsoids, first obtain the scales of all Gaussian ellipsoids and take out the minimum value τ in the scale of each Gaussian ellipsoid i , the scale loss of the Gaussian ellipsoid The function is expressed as

[0032]

[0033] Furthermore, in the above step S2.1, the projection process of the Gaussian mean is:

[0034] First, assume that [x w ,y w ,z w ] T is the mean of the Gaussian ellipsoid in the world coordinate system. The mean is transformed to the camera coordinate system. The transformation formula is:

[0035] [x c ,y c ,z c ] T =R[x w ,y w ,z w ] T +t

[0036] Among them, [x c ,y c ,z c ] T is the coordinate of the center (mean) of the Gaussian ellipsoid in the camera coordinate system, (R, t) is the position of the camera; T represents transposition;

[0037] Then, calculate the perspective projection matrix M persp

[0038]

[0039] Among them, f x 、f y are the focal lengths of the camera in the horizontal and vertical directions of the image, respectively; (c x 、c y ) is the principal point coordinate of the camera; W and H are the width and height of the image respectively; f and n are the far plane and near plane of the viewing cone respectively;

[0040] Finally, the Gaussian mean is transformed into a homogeneous coordinate system and combined with the perspective projection matrix M persp Multiply them to get the coordinates of the Gaussian mean in the NDC coordinate system (x ndc ,y ndc ,z ndc ); At this time, there is x ndc ∈[-1,1],y ndc ∈[-1,1],z ndc ∈[0,1]; take the first two dimensions of the NDC coordinates (x ndc ,y ndc ) and scale it to W and H to obtain the coordinates of the Gaussian mean in the image plane.

[0041] Furthermore, in the above step S2.1, the projection process of the Gaussian variance is:

[0042] First, assume that [x w ,y w ,z w ] T is the mean of the Gaussian ellipsoid in the world coordinate system. The mean is transformed to the camera coordinate system. The transformation formula is:

[0043] [x c ,y c ,z c ] T =R[x w ,y w ,z w ] T +t

[0044] Among them, [xc ,y c ,z c ] T is the coordinate of the center (mean) of the Gaussian ellipsoid in the camera coordinate system, (R, t) is the position of the camera; T represents transposition;

[0045] Then, the Jacobian matrix J is calculated based on the mean;

[0046]

[0047] Assume that the variance of the Gaussian ellipsoid in the world coordinate system is Σ 3D , transform the covariance using the Jacobian matrix and the camera rotation, Σ'=JRΣ 3D R T J; where R is the rotation of the camera;

[0048] Finally, take the first two rows and columns of Σ' to get the projection of the variance on the two-dimensional plane Σ 2D .

[0049] Furthermore, the above step S2.2 includes the following sub-steps:

[0050] First, arrange the M Gaussian ellipsoids from near to far according to their distance from the camera plane;

[0051] Then, calculate each Gaussian ellipsoid for pixel p i Transparency of β i,j ;

[0052]

[0053] Among them, β' j is the transparency of the Gaussian ellipsoid itself; μ j is the projection position of the mean of the Gaussian ellipsoid on the image plane; Σ j is the variance of the Gaussian ellipsoid projection on the image plane; β i,j Represents the jth Gaussian ellipsoid for pixel p i transparency;

[0054] Finally, we get the Gaussian ellipsoid j for each pixel p i Transparency of β i,j Then, according to the weight w i,j For each Gaussian ellipsoid color C j Perform weighted averaging and calculate each pixel p i Final Color

[0055]

[0056] Furthermore, the above step S3 includes the following sub-steps:

[0057] S3.1. Randomly select one Gaussian ellipsoid visible in the current image and send out a ray starting from the optical center and pointing to the center of the Gaussian ellipsoid; uniformly sample n1 points on this ray to achieve coarse sampling so that the sampling points cover the entire scene; then calculate the distance d from the Gaussian center point to the camera center, uniformly sample n2 points within the distance range of [0.9d, 1.1d] to achieve fine sampling, which is mainly used to restore the geometric structure of the scene; each ray generates (n1+n2) sampling points; for the (n1+n2) sampling points, estimate the color and SDF value of the sampling points to obtain the color of the entire ray;

[0058] S3.2, repeat step S3.1 m times, that is, select m Gaussian ellipsoids within the field of view as sampling guides, and then obtain m rays, a total of m*(n1+n2) sampling points; add the losses of these rays and sampling points together to obtain the final Gaussian ray loss and Gaussian SDF loss

[0059] S3.3, randomly sample K rays on each image, the color of each ray being the color of the pixel at the sampling point of the ray;

[0060] S3.4. On each ray, randomly sample Q 3D points in the order from near to far from the camera center, input the 3D points into the neural network, and output the SDF value corresponding to each 3D point And the color of each 3D point The estimated color of each ray is calculated by the method in step S3.1

[0061] Next, calculate the estimated color and the true value color C j Radiation field ray loss between

[0062]

[0063] S3.5. Randomly select a 3D point p in the laser radar point cloud, the distance between the radar point and the laser radar center is d'; emit a ray pointing to p from the radar center, randomly sample Q 3D points on the ray in the order from near to far from the laser radar center, input the 3D points into the neural network, and output the SDF value corresponding to each 3D point Also calculate the distance d from each 3D point to the center of the laser radar i ; Calculate different SDF errors according to the different positions of the 3D points;

[0064] S3.6, repeat step S3.5 r times, that is, select r laser radar points, generate r rays, and add the losses of these rays together to obtain the radiation field SDF loss

[0065] Furthermore, the above step S3.1 specifically includes:

[0066] The (n1+n2) 3D points are input into a neural network, which outputs the SDF value and color of each 3D point. The 3D points are arranged in order from near to far from the camera center. Assuming that the SDF value of a certain point is The color value is The weight w of each 3D point is obtained by the SDF value i , the formula is

[0067]

[0068] in, is the SDF value of two adjacent 3D sampling points; Sig(*) represents the Sigmoid function; after obtaining the weight, the weighted average of the colors can be used to obtain the color of the entire ray.

[0069]

[0070] Calculate the Gaussian ray loss between the estimated ray color and the true ray color C

[0071] For the sampling points within the distance range of the ray [0.9d, 1.1d], that is, n2 fine sampling points, the error between the estimated SDF value and the true SDF value of each point is additionally calculated, where the true value of SDF s i is the distance from each point to the center of the Gaussian ellipsoid; if the sampling point is between the camera optical center and the Gaussian center, then s i is a positive number; if the sampling point is outside the Gaussian ellipsoid, then s i is a negative number;

[0072] Through the above definition, the SDF value is divided into positive and negative, that is, the direction of the object surface is specified: at points outside the object surface, the SDF is positive; at points inside the object surface, the SDF is negative; the Gaussian SDF loss between the estimated SDF value and the true value

[0073] Furthermore, in the above step S4, all the losses in step S2 and step S3 are added together to obtain the final loss

[0074]

[0075] Among them, λ1, λ2, λ3, λ4, λ5, λ6 are weights;

[0076] For rendering loss; is the scale loss; is the Gaussian ray loss; is Gaussian SDF loss; is the radiation field ray loss; is the radiation field SDF loss.

[0077] Furthermore, in the above step S3.5, if the 3D point is far from the radar point, that is, |d i -d'|>0.3, then the SDF error of this point is

[0078] If the 3D point is close to the radar point, that is, |d i -d'|≤0.3, then the SDF error of this point is Among them, (d'-d i ) is the true value of the SDF at that point.

[0079] Furthermore, the above step S8 includes the following sub-steps:

[0080] S8.1. First, preset the resolution of the Mesh and divide the entire scene into Y segments along the three coordinate axes. 3 Individual voxel blocks;

[0081] S8.2, extract the coordinates of the midpoint of each voxel block, input these points into the neural network, and obtain the SDF value of each point;

[0082] S8.3. Using the marching tetrahedron algorithm according to the SDF value of the point, a three-dimensional reconstructed mesh model of the entire scene is obtained.

[0083] Due to the adoption of the above-mentioned technical solution, the present invention has the following advantages:

[0084] This complex scene 3D reconstruction method based on multi-source fusion integrates neural radiance field (NeRF) and three-dimensional Gaussian sputtering (3DGS), and NeRF and 3D GS guide each other; it can implicitly associate multimodal data in space without explicit cross-modal feature matching; it can directly extract a three-dimensional mesh model (Mesh) for better use in other downstream tasks; at the same time, it can obtain a new perspective rendering result of the scene and a mesh model of the scene; in the process of 3D reconstruction, under the condition of a given initial pose, it can jointly optimize the camera and lidar poses and the implicit expression of the scene to obtain more accurate pose and scene 3D reconstruction results, achieve higher reconstruction accuracy, and have good promotion and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 It is a flow chart of a complex scene 3D reconstruction method based on multi-source fusion of the present invention;

[0086] Figure 2 is a comparison diagram of reconstruction results on a single object in one embodiment of the present invention;

[0087] Figure 3 is a comparison diagram of reconstruction results on a single object in the second embodiment of the present invention;

[0088] Figure 4 This is a comparison diagram of reconstruction results of outdoor scenes in autonomous driving in the third embodiment of the present invention;

[0089] Figure 5 This is a comparison diagram of reconstruction results of outdoor scenes in autonomous driving in the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0090] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0091] like Figure 1 As shown, the complex scene 3D reconstruction method based on multi-source fusion includes the following specific steps:

[0092] S1. Collecting multiple frames of continuous point cloud data of the target scene through a laser radar; using the point cloud data to perform pose estimation to obtain laser radar pose information; collecting multiple frames of continuous image data of the target scene through a camera; using the image data to perform pose estimation to obtain camera pose information;

[0093] S2, based on the camera image data and camera pose information in step S1, calculating a loss function related to the three-dimensional ellipsoid, the loss function is used to train the properties of the three-dimensional ellipsoid and optimize the camera pose, the properties of the three-dimensional ellipsoid include: color, position (mean), transparency, rotation, scale, wherein the rotation and scale are combined to obtain the variance of the three-dimensional Gaussian ellipsoid; comprising the following sub-steps:

[0094] S2.1. Randomly select an image, and project N Gaussian ellipsoids of the target scene to the image coordinates according to the position and posture of the image. The projection process includes projecting the mean and variance of the Gaussian ellipsoids respectively, obtaining the coordinates of the Gaussian mean under the image plane and the projection of the variance on the two-dimensional plane, and determining the projected Gaussian data;

[0095] (1) The projection process of the Gaussian mean is:

[0096] First, assume that [x w ,y w ,z w ] T is the mean of the Gaussian ellipsoid in the world coordinate system. The mean is transformed to the camera coordinate system. The transformation formula is:

[0097] [x c ,y c ,z c ] T =R[x w ,y w ,z w ] T +t

[0098] Among them, [x c ,y c ,z c ] T is the coordinate of the center (mean) of the Gaussian ellipsoid in the camera coordinate system, (R, t) is the position of the camera; T represents transposition;

[0099] Then, calculate the perspective projection matrix M persp

[0100]

[0101] Among them, f x 、f y are the focal lengths of the camera in the horizontal and vertical directions of the image, respectively; (c x 、c y ) is the principal point coordinate of the camera; W and H are the width and height of the image respectively; f and n are the far plane and near plane of the viewing cone respectively;

[0102] Finally, the Gaussian mean is transformed into a homogeneous coordinate system and combined with the perspective projection matrix Mpersp Multiply them to get the coordinates of the Gaussian mean in the NDC coordinate system (x ndc ,y ndc ,z ndc ); At this time, there is x ndc ∈[-1,1],y ndc ∈[-1,1],z ndc ∈[0,1]; take the first two dimensions of the NDC coordinates (x ndc ,y ndc ) and scale it to W and H to obtain the coordinates of the Gaussian mean in the image plane;

[0103] (2) The projection process of Gaussian variance is:

[0104] First, assume that [x w ,y w ,z w ] T is the mean of the Gaussian ellipsoid in the world coordinate system. The mean is transformed to the camera coordinate system. The transformation formula is:

[0105] [x c ,y c ,z c ] T =R[x w ,y w ,z w ] T +t

[0106] Among them, [x c ,y c ,z c ] T is the coordinate of the center (mean) of the Gaussian ellipsoid in the camera coordinate system, (R, t) is the position of the camera; T represents transposition;

[0107] Then, the Jacobian matrix J is calculated based on the mean;

[0108]

[0109] Assume that the variance of the Gaussian ellipsoid in the world coordinate system is Σ 3D , transform the covariance using the Jacobian matrix and the camera rotation, Σ'=JRΣ 3D R T J; where R is the rotation of the camera;

[0110] Finally, take the first two rows and columns of Σ' to get the projection of the variance on the two-dimensional plane Σ 2D ;

[0111] S2.2. Based on the projected Gaussian data, a new image is rendered. Each pixel p on the imagei The rendered color is the weighted average of the colors of M Gaussian ellipsoids, and N ≥ M. This is because the area of ​​each Gaussian projection is limited, and only the Gaussian ellipsoid projected to the pixel position is used when rendering each pixel; it includes the following sub-steps:

[0112] First, arrange the M Gaussian ellipsoids from near to far according to their distance from the camera plane;

[0113] Then, calculate each Gaussian ellipsoid for pixel p i Transparency of β i,j ;

[0114]

[0115] Among them, β' j is the transparency of the Gaussian ellipsoid itself; μ j is the projection position of the mean of the Gaussian ellipsoid on the image plane; Σ j is the variance of the Gaussian ellipsoid projection on the image plane; β i,j Represents the jth Gaussian ellipsoid for pixel p i transparency;

[0116] Finally, we get the Gaussian ellipsoid j for each pixel p i Transparency of β i,j Then, according to the weight w i,j For each Gaussian ellipsoid color C j Perform weighted averaging and calculate each pixel p i Final Color

[0117]

[0118] S2.3. Obtaining a rendered image After that, the rendered image and the real image are calculated Rendering loss

[0119]

[0120] Among them, SSIM(a,b) represents the structural similarity of two images; λ ssim Is a hyperparameter used to balance the two errors. Generally, λ ssim =0.2;

[0121] S2.4. Assuming there are N Gaussian ellipsoids, first obtain the scales of all Gaussian ellipsoids and take out the minimum value τ in the scale of each Gaussian ellipsoid i , the scale loss of the Gaussian ellipsoid The function is expressed as

[0122]

[0123] The purpose of this loss is to minimize the minimum scale of the Gaussian ellipsoid, because each Gaussian ellipsoid has three scales. By minimizing the minimum scale, the ellipsoid can be made as flat as possible, that is, the ellipsoid can be turned into a "disc", so that the Gaussian fits tightly to the surface of the scene object.

[0124] S3, based on the camera image data and the laser radar point cloud data in step S1, as well as the camera pose and the laser radar pose, calculating the loss function related to the neural radiation field; comprising the following sub-steps:

[0125] S3.1. Randomly select one Gaussian ellipsoid visible in the current image and send out a ray starting from the optical center and pointing to the center of the Gaussian ellipsoid; uniformly sample n1 points on this ray to achieve coarse sampling so that the sampling points cover the entire scene; then calculate the distance d from the Gaussian center point to the camera center, uniformly sample n2 points in the distance range of [0.9d, 1.1d] to achieve fine sampling, which is mainly used to restore the geometric structure of the scene; in this way, each ray will generate (n1+n2) sampling points; the number of sampling points n1 and n2 is generally selected as 2 to the power of N, preferably, n1 is generally selected as 256 or 512, and n2 is generally selected as 64 or 128; for the (n1+n2) sampling points, estimate the color and SDF value of the sampling points to obtain the color of the entire ray, and the specific operations include:

[0126] The (n1+n2) 3D points are input into a neural network, which outputs the SDF value and color of each 3D point. The 3D points are arranged in order from near to far from the camera center. Assuming that the SDF value of a certain point is The color value is The weight w of each 3D point is obtained by the SDF value i , the formula is

[0127]

[0128]

[0129] in, is the SDF value of two adjacent 3D sampling points; Sig(*) represents the Sigmoid function; after obtaining the weight, the weighted average of the colors can be used to obtain the color of the entire ray.

[0130]

[0131] Calculate the Gaussian ray loss between the estimated ray color and the true ray color C

[0132] For the sampling points within the distance range of the ray [0.9d, 1.1d], that is, n2 fine sampling points, the error between the estimated SDF value and the true SDF value of each point is additionally calculated, where the true value of SDF s i is the distance from each point to the center of the Gaussian ellipsoid; if the sampling point is between the camera optical center and the Gaussian center, then s i is a positive number; if the sampling point is outside the Gaussian ellipsoid (that is, the Gaussian ellipsoid is between the sampling point and the camera optical center), then s i is a negative number;

[0133] Through the above definition, the SDF value is divided into positive and negative, that is, the direction of the object surface is specified: at points outside the object surface, the SDF is positive; at points inside the object surface, the SDF is negative; the Gaussian SDF loss between the estimated SDF value and the true value

[0134] S3.2, repeat step S3.1 m times, that is, select m Gaussian ellipsoids within the field of view as sampling guides, and then obtain m rays, a total of m*(n1+n2) sampling points; add the losses of these rays and sampling points together to obtain the final Gaussian ray loss and Gaussian SDF loss

[0135] S3.3, randomly sample K rays on each image, the color of each ray being the color of the pixel at the sampling point of the ray;

[0136] S3.4. On each ray, randomly sample Q 3D points in the order from near to far from the camera center, input the 3D points into the neural network, and output the SDF value corresponding to each 3D point And the color of each 3D point The estimated color of each ray is calculated by the method in step S3.1

[0137] Next, calculate the estimated color and the true value color C j Radiation field ray loss between

[0138]

[0139] S3.5. Randomly select a 3D point p in the laser radar point cloud, the distance between the radar point and the laser radar center is d'; emit a ray pointing to p from the radar center, randomly sample Q 3D points on the ray in the order from near to far from the laser radar center, input the 3D points into the neural network, and output the SDF value corresponding to each 3D point Also calculate the distance d from each 3D point to the center of the laser radar i ; Calculate different SDF errors according to the different positions of the 3D points;

[0140] If the 3D point is far from the radar point, that is, |d i -d'|>0.3, then the SDF error of this point is

[0141] If the 3D point is close to the radar point, that is, |d i -d'|≤0.3, then the SDF error of this point is Among them, (d'-d i ) is the true value of the SDF at that point;

[0142] S3.6, repeat step S3.5 r times, that is, select r laser radar points, generate r rays, and add the losses of these rays together to obtain the radiation field SDF loss

[0143] S4. Add all the losses in step S2 and step S3 together to get the final loss

[0144] Among them, λ1, λ2, λ3, λ4, λ5, and λ6 are weights used to balance different types of losses;

[0145] There are different types of losses. For rendering loss; is the scale loss; is the Gaussian ray loss; is Gaussian SDF loss; is the radiation field ray loss; is the radiation field SDF loss;

[0146] The final loss Perform back propagation, i.e. optimize the 3D Gaussian ellipsoid, the neural radiance field, and the camera and lidar poses;

[0147] S5. Dynamically increase the Gaussian ellipsoid according to the gradient of the final loss propagated to the mean of the Gaussian ellipsoid;

[0148] If the gradient of a Gaussian mean is large and the scale of the Gaussian is small, the Gaussian ellipsoid is cloned once and the cloned Gaussian ellipsoid is moved according to the gradient of the mean;

[0149] If the gradient of a Gaussian mean is large and the scale of the Gaussian is large, the Gaussian ellipsoid is split into two smaller Gaussian ellipsoids, each with a scale of half the original scale, and one of the sub-Gaussians is moved according to the gradient of the mean;

[0150] S6. If the transparency of a Gaussian ellipsoid is lower than 0.001, that is, the Gaussian ellipsoid is very transparent and has almost no effect on rendering, the Gaussian ellipsoid is directly deleted;

[0151] S7, repeating the above steps S2 to S6 at least tens of thousands of times, completing the training, and obtaining a three-dimensional expression of the entire scene;

[0152] S8, extracting a three-dimensional reconstruction result of the target scene according to the trained neural radiation field, including the following sub-steps:

[0153] S8.1. First, preset the resolution of the Mesh, that is, divide the entire scene into Y segments along the three coordinate axes. 3 The higher the resolution, the finer the result, but the slower the calculation speed and the larger the file size. In general, 256 is selected, which means that the whole scene is divided into 256 segments along the three coordinate axes. 3 Individual voxel blocks;

[0154] S8.2, extract the coordinates of the midpoint of each voxel block, input these points into the neural network, and obtain the SDF value of each point;

[0155] S8.3. Using the Marching Cube algorithm according to the SDF value of the point, a three-dimensional reconstructed mesh model (Mesh) of the entire scene is obtained.

[0156] from Figure 2 , 3 It can be seen from Figures 4 and 5 that the complex scene 3D reconstruction method based on multi-source fusion of the present invention can relatively completely reconstruct the entire 3D scene with high reconstruction accuracy.

[0157] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Those skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents; these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A complex scene 3D reconstruction method based on multi-source fusion, Its characteristics are: It includes the following steps: S1. Collecting laser radar point cloud data and camera image data of the target scene, using the laser radar point cloud data to perform pose estimation to obtain laser radar pose information; using the camera image data to perform pose estimation to obtain camera pose information; S2. Based on the camera image data and camera pose information in step S1, a loss function related to the three-dimensional ellipsoid is calculated, where the loss function is used to train the properties of the three-dimensional ellipsoid and optimize the camera pose. The properties of the three-dimensional ellipsoid include: color, position, transparency, rotation, and scale. The rotation and scale are combined to obtain the variance of the three-dimensional Gaussian ellipsoid. S3, based on the camera image data and the lidar point cloud data in step S1, as well as the camera and lidar poses, calculating a loss function related to the neural radiation field; S4, adding all the losses in step S2 and step S3 together to obtain the final loss; back-propagating the final loss to optimize the three-dimensional Gaussian ellipsoid, the neural radiation field, and the camera and lidar poses; S5. Dynamically increase the Gaussian ellipsoid according to the gradient of the final loss propagated to the mean of the Gaussian ellipsoid; S6. If the transparency of a Gaussian ellipsoid is lower than 0.001, directly delete the Gaussian ellipsoid; S7, repeating the above steps S2 to S6 at least tens of thousands of times, completing the training, and obtaining a three-dimensional expression of the entire scene; S8. Extract the three-dimensional reconstruction result of the scene based on the trained neural radiation field.

2. The method for 3D reconstruction of complex scenes based on multi-source fusion according to claim 1, Its characteristics are: The step S2 comprises the following sub-steps: S2.

1. Randomly select an image, and project N Gaussian ellipsoids of the target scene to the image coordinates according to the position and posture of the image. The projection process includes projecting the mean and variance of the Gaussian ellipsoids respectively, obtaining the coordinates of the Gaussian mean under the image plane and the projection of the variance on the two-dimensional plane, and determining the projected Gaussian data; S2.

2. Render a new image based on the projected Gaussian data. Each pixel p on the image i The rendered color is the weighted average of the colors of M Gaussian ellipsoids, and N ≥ M; S2.

3. Obtaining a rendered image After that, the rendered image and the real image are calculated Error Among them, SSIM(a,b) represents the structural similarity of two images; λ ssim is a hyperparameter used to balance the two errors; S2.

4. Assuming there are N Gaussian ellipsoids, first obtain the scales of all Gaussian ellipsoids and take out the minimum value τ in the scale of each Gaussian ellipsoid i , the loss function of the Gaussian ellipsoid scale is expressed as 3. The method for 3D reconstruction of complex scenes based on multi-source fusion according to claim 2, Its characteristics are: In step S2.1, the projection process of the Gaussian mean is: First, assume that [x w ,y w ,z w ] T is the mean of the Gaussian ellipsoid in the world coordinate system. The mean is transformed to the camera coordinate system. The transformation formula is: [x c ,y c ,z c ] T =R[x w ,y w ,z w ] T +t Among them, [x c ,y c ,z c ] T is the coordinate of the center (mean) of the Gaussian ellipsoid in the camera coordinate system, (R, t) is the position of the camera; T represents transposition; Then, calculate the perspective projection matrix M persp Among them, f x 、f y are the focal lengths of the camera in the horizontal and vertical directions of the image, respectively; (c x 、c y ) is the principal point coordinate of the camera; W and H are the width and height of the image respectively; f and n are the far plane and near plane of the viewing cone respectively; Finally, the Gaussian mean is transformed into a homogeneous coordinate system and combined with the perspective projection matrix M persp Multiply them to get the coordinates of the Gaussian mean in the NDC coordinate system (x ndc ,y ndc ,z ndc ); At this time, there is x ndc ∈[-1,1],y ndc ∈[-1,1],z ndc ∈[0,1]; take the first two dimensions of the NDC coordinates (x ndc ,y ndc ) and scale it to W and H to obtain the coordinates of the Gaussian mean in the image plane.

4. The method for 3D reconstruction of complex scenes based on multi-source fusion according to claim 2, Its characteristics are: In step S2.1, the projection process of Gaussian variance is: First, assume that [x w ,y w ,z w ] T is the mean of the Gaussian ellipsoid in the world coordinate system. The mean is transformed to the camera coordinate system. The transformation formula is: [x c ,y c ,z c ] T =R[x w ,y w ,z w ] T +t Among them, [x c ,y c ,z c ] T is the coordinate of the center (mean) of the Gaussian ellipsoid in the camera coordinate system, (R, t) is the position of the camera; T represents transposition; Then, the Jacobian matrix J is calculated based on the mean; Assume that the variance of the Gaussian ellipsoid in the world coordinate system is Σ 3D , transform the covariance using the Jacobian matrix and the camera rotation, Σ'=JRΣ 3D R T J; where R is the rotation of the camera; Finally, take the first two rows and columns of Σ' to get the projection of the variance on the two-dimensional plane Σ 2D .

5. The method for 3D reconstruction of complex scenes based on multi-source fusion according to claim 2, Its characteristics are: The step S2.2 includes the following sub-steps: First, arrange the M Gaussian ellipsoids from near to far according to their distance from the camera plane; Then, calculate each Gaussian ellipsoid for pixel p i Transparency of β i,j ; Among them, β' j is the transparency of the Gaussian ellipsoid itself; μ j is the projection position of the mean of the Gaussian ellipsoid on the image plane; Σ j is the variance of the Gaussian ellipsoid projection on the image plane; β i,j Represents the jth Gaussian ellipsoid for pixel p i transparency; Finally, we get the Gaussian ellipsoid j for each pixel p i Transparency of β i,j Then, according to the weight w i,j For each Gaussian ellipsoid color C j Perform weighted averaging and calculate each pixel p i Final Color 6. The method for 3D reconstruction of complex scenes based on multi-source fusion according to claim 1, Its characteristics are: The step S3 comprises the following sub-steps: S3.

1. Randomly select one Gaussian ellipsoid visible in the current image and send out a ray starting from the optical center and pointing to the center of the Gaussian ellipsoid. Uniformly sample n 1 points to achieve coarse sampling so that the sampling points cover the entire scene; then, calculate the distance d from the Gaussian center point to the camera center, and uniformly sample n in the distance range of [0.9d, 1.1d] 2 points to achieve fine sampling, which is mainly used to restore the geometric structure of the scene; each ray generates (n 1 +n 2 ) sampling points; for the (n 1 +n 2 ) sampling points, estimating the color and SDF value of the sampling points to obtain the color of the entire ray; S3.2, repeat step S3.1 m times, i.e. select m Gaussian ellipsoids within the field of view as sampling guides, and then obtain m rays, a total of m*(n 1 +n 2 ) sampling points; add the losses of these rays and sampling points together to get the final Gaussian ray loss and Gaussian SDF loss S3.3, randomly sample K rays on each image, the color of each ray being the color of the pixel at the sampling point of the ray; S3.

4. On each ray, randomly sample Q 3D points in the order from near to far from the camera center, input the 3D points into the neural network, and output the SDF value corresponding to each 3D point And the color of each 3D point The estimated color of each ray is calculated by the method in step S3.1 Next, calculate the estimated color and the true value color C j The error between S3.

5. Randomly select a 3D point p in the laser radar point cloud, the distance between the radar point and the laser radar center is d'; emit a ray pointing to p from the radar center, randomly sample Q 3D points on the ray in the order from near to far from the laser radar center, input the 3D points into the neural network, and output the SDF value corresponding to each 3D point Also calculate the distance d from each 3D point to the center of the laser radar i ; Calculate different SDF errors according to the different positions of the 3D points; S3.6, repeat step S3.5 r times, that is, select r laser radar points, generate r rays, and add the losses of these rays together to get the final loss 7. The method for 3D reconstruction of complex scenes based on multi-source fusion according to claim 6, Its characteristics are: The step S3.1 comprises: The (n 1 +n 2 ) 3D points are input into a neural network, which outputs the SDF value and color of each 3D point; these 3D points are arranged in order from near to far from the camera center. Assuming that the SDF value of a point is The color value is The weight w of each 3D point is obtained by the SDF value i , the formula is in, is the SDF value of two adjacent 3D sampling points; Sig(*) represents the Sigmoid function; after obtaining the weight, the weighted average of the colors can be used to obtain the color of the entire ray. Calculate the error between the estimated ray color and the true ray color C For sampling points within the distance range of the ray [0.9d, 1.1d], that is, n 2 The error between the estimated SDF value and the true SDF value at each point is calculated, where the true value of SDF is s i is the distance from each point to the center of the Gaussian ellipsoid; if the sampling point is between the camera optical center and the Gaussian center, then s i is a positive number; if the sampling point is outside the Gaussian ellipsoid, then s i is a negative number; Through the above definition, the SDF value is divided into positive and negative, that is, the direction of the object surface is specified: at a point outside the object surface, the SDF is positive; at a point inside the object surface, the SDF is negative; the error between the estimated SDF value and the true value is 8. The method for 3D reconstruction of complex scenes based on multi-source fusion according to claim 6, Its characteristics are: In step S3.5, if the 3D point is far from the radar point, that is, |d i -d'|>0.3, then the SDF error of this point is If the 3D point is close to the radar point, that is, |d i -d'|≤0.3, then the SDF error of this point is Among them, (d'-d i ) is the true value of the SDF at that point.

9. The method for 3D reconstruction of complex scenes based on multi-source fusion according to claim 1, Its characteristics are: The step S8 comprises the following sub-steps: S8.

1. First, preset the resolution of the Mesh and divide the entire scene into Y segments along the three coordinate axes. 3 Individual voxel blocks; S8.2, extract the coordinates of the midpoint of each voxel block, input these points into the neural network, and obtain the SDF value of each point; S8.

3. Using the marching tetrahedron algorithm according to the SDF value of the point, a three-dimensional reconstructed mesh model of the entire scene is obtained.

Citation Information

Patent Citations

  • Three-dimensional reconstruction method, system and device in low-illumination scene and medium

    CN116824051A

  • Shielding removing method based on nerve radiation field

    CN116977360A

  • Multi-view three-dimensional reconstruction method based on deep residual error and neural implicit surface learning

    CN117830520A

  • Point cloud processing method and device, medium and electronic equipment

    CN118314055A

Cited By

  • Scene three-dimensional reconstruction method based on camera and laser radar fusion and related device

    CN120088403A

  • Reinforcement learning adaptive multi-mode SLAM method based on 4D Gaussian splashing

    CN120141447A

  • Fire scene three-dimensional reconstruction method and device and electronic equipment

    CN120431245A

  • Multi-feature 3D (three-dimensional) Gaussian reconstruction method based on laser vision

    CN121190660A

  • A multi-feature three-dimensional reconstruction 3D Gaussian method based on laser vision

    CN121190660B