A self-supervised three-dimensional scene mapping method based on multi-view pose self-optimization

Abstract

The application discloses a kind of self-supervised three-dimensional scene mapping methods based on multi-view pose self-optimization, specifically includes: given RGB-D closed loop video stream in any adjacent data, corresponding two-dimensional matching points are extracted and mapped to three-dimensional space, estimate initial pose, then according to initial pose, RGB-D image is mapped to three-dimensional space to obtain point cloud data;Any frame point cloud is rendered to adjacent frame view, and the deviation of the rendered RGB-D image and the original image of adjacent view is minimized to optimize interframe pose;Then any frame point cloud is looped along the camera path and sequentially rendered in reverse, and the deviation of the reverse rendered image and the original image of adjacent view is minimized to complete global pose optimization.The method provided by the application provides an efficient and low-cost solution, which can complete high-precision multi-view point cloud registration and reconstruction without relying on complex hardware and real pose data.

Description

Technical Field

[0001] This invention belongs to the field of 3D visual point cloud registration technology, and particularly relates to a self-supervised 3D scene mapping method based on multi-view pose self-optimization. Background Technology

[0002] High-precision 3D reconstruction of complete indoor scenes has always been a challenging task. Traditional 3D reconstruction methods rely on sensor devices such as Time-of-Flight (ToF), stereo vision, structured light, and 3D radar. With the emergence of RGB-D sensing devices such as Kinect, these devices have become effective alternatives to traditional 3D sensing devices due to their low cost and high-quality reconstruction capabilities. Although depth cameras can easily reconstruct single-viewpoint clouds of objects, the accuracy and reliability of reconstructing complete indoor scenes have not yet reached ideal levels. Therefore, to obtain a complete 3D map, global registration of 3D point cloud data from different viewpoints is necessary.

[0003] Currently, many effective point cloud registration methods focus on the registration of paired point clouds, which typically estimate pose transformation by extracting corresponding points. Traditional registration algorithms usually rely on feature extractors (such as PFH and FPFH) and pose estimators (such as RANSAC) to achieve the registration of paired point clouds. However, the feature extraction capabilities of traditional methods are often limited, making it difficult to handle diverse data types, which poses a challenge for their application in complex industrial scenarios. As the complexity of point cloud data increases, traditional methods often result in low registration accuracy and insufficient stability when processing large-scale, noisy point cloud data, making it difficult to meet the high-precision and high-robustness requirements of industrial applications.

[0004] Deep learning methods, by training more robust and accurate feature extractors, can automatically learn complex spatial features from large amounts of data, thereby obtaining more precise correspondences and significantly improving the accuracy of pose estimation. Compared with traditional methods, deep learning methods have shown significant advantages in extracting complex geometric features, processing large-scale point cloud data, and dealing with noise and occlusion. However, although deep learning methods can improve accuracy in pairwise point cloud registration, their main problem remains: in registration tasks with complete scene data, pairwise point cloud registration requires the camera to collect data multiple times along complex trajectories, leading to a gradual increase in accumulated drift error (odometry drift), which ultimately affects the accuracy of the overall registration result.

[0005] Therefore, research on scene mapping has recognized the importance of global registration optimization. Common solutions involve two stages: first, applying pairwise point cloud registration to estimate the poses between adjacent views of the point cloud; then, performing global optimization on the poses of all views. However, a common drawback of this process is that the global pose deviation is usually highly correlated with the initial pose. Significant initial deviations often affect subsequent optimization results, preventing convergence to the ideal solution and impacting the final point cloud reconstruction accuracy. In recent years, researchers have proposed deep learning-based optimization methods that transform the global point cloud registration problem into an iterative weighted least squares (IRLS) problem. By continuously optimizing the estimation of neighboring and absolute poses, this method can gradually improve registration accuracy and reduce the negative impact of odometry drift. However, this method relies on real pose labels to guide model feature training. Obtaining real pose labels typically requires high-precision external sensors for auxiliary measurement, which further limits the practical application scope and scalability of the method, resulting in high dataset production costs and limited generalization ability. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention proposes a self-supervised 3D scene mapping method based on multi-view pose self-optimization. Specifically, for a given closed-loop continuous RGB-D image, the corresponding keypoints of the RGB images of neighboring frames are first calculated and mapped to 3D space. Then, the initial pose of the neighboring frames is estimated using 3D point pairs filtered by spatial compatibility. For the RGB and depth images of a given view, the initial pose is used to render it to the next neighboring view to calculate the pose deviation. Furthermore, the closed-loop accumulated pose is used to back-render the RGB and depth images of the given view to the previous neighboring view to evaluate the global pose deviation. The method proposed in this invention is inspired by neural radiation fields, and the view rendering method better constrains pose estimation. It aims to achieve pose deviation characterization by comparing the differences between neighboring view images and the rendered image without relying on real pose data, thereby solving the problems faced by traditional point cloud registration methods when dealing with self-similar structures and complex scenes.

[0007] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution:

[0008] A self-supervised 3D scene mapping method based on multi-view pose self-optimization, which specifically includes:

[0009] S1. Given any adjacent data in an RGB-D closed-loop video stream, extract the corresponding two-dimensional matching points and map them to three-dimensional space to estimate the initial pose.

[0010] S2. Under the initial pose, map the RGB-D image to the three-dimensional space to obtain point cloud data; render the point cloud of any frame to the view of the adjacent frame to obtain the rendered RGB-D image, and minimize the deviation between the rendered RGB-D image and the original image of the adjacent view to optimize the inter-frame pose.

[0011] S3. Then, the point cloud of any frame is rendered backward along the camera path to the viewpoint of the adjacent frame to obtain the RGB-D image after inversion rendering. The deviation between the image after inversion rendering and the original image of the adjacent viewpoint is minimized to complete the global pose optimization.

[0012] Furthermore, step S1 specifically includes:

[0013] S11. Given RGB-D images from multiple perspectives Camera Insider Among them I i D represents the color information of the three channels. i Representing depth information, N represents the number of input RGB-D images, and i represents the input RGB-D image of the i-th frame; for any adjacent RGB images at viewpoints a and b, ORB features are selected for extraction to obtain 2D matching points {O}. 2a O 2b};

[0014] S12, Use a depth map aligned with the RGB image and camera intrinsic K a ,K b Mapping the 2D matching points from adjacent viewpoints a and b to 3D space yields the 3D matching point {O}. 3a O 3b Then, based on spatial compatibility filtering, high-quality 3D matching points {H} are obtained. 3a H 3b The formula is expressed as:

[0015] {H 3a H 3b}=|d(a i ,a j )-d(b i ,b j )| <d th ;

[0016] Among them, (a i ,b i ) and (a j ,b j ) is {O 3a O 3b For any two pairs of 3D matching points in}, d(·,·) is the Euclidean distance, d th Distance threshold;

[0017] S13. Obtain high-quality three-dimensional matching points {H} 3a H 3b Then, the initial pose between adjacent viewpoints is estimated using the pose estimator RANSAC.

[0018] Furthermore, step S2 specifically includes:

[0019] S21. Given a set of RGB-D images under closed-loop continuous multi-view conditions. Camera internal parameters Where N is the number of input views, I i D represents the color information of the three channels. i Representing depth information, K i Intrinsic parameters for each camera; using camera intrinsic parameters to convert RGB-D images (I i D i The point clouds P of each view are obtained by back-projection. i Specifically: First, a 3D mesh (3×H×W) is created. i Mapped to a 3D screen coordinate system, and then using the camera intrinsic parameter K. i The point cloud is obtained by inverse transformation of the screen coordinate system to the camera coordinate system, and then based on I... i and D i Pixel consistency maps RGB onto the point cloud, as expressed by the formula:

[0020] P i =p -1 (I i D i ,K i );

[0021] Where, p -1 (·,·,·) denotes the back projection function;

[0022] S22. Given a set of initially aligned point clouds The pose between neighboring views is optimized by minimizing the deviation between the rendered image and the original image of the point cloud from neighboring views as the local optimization objective function.

[0023] S23. Calculate the mask of the rendered image during the rendering process, and let the effective pixels that are not occluded participate in the deviation calculation. Improve the objective function to alleviate the pixel loss between the rendered image and the original image caused by object occlusion due to the difference in viewpoint, which affects the accuracy of pose constraint.

[0024] More specifically, the local optimization objective function in step S22 is as follows:

[0025]

[0026] in, For the RGB image rendering process, This refers to the depth image rendering process; the rendering process specifically involves using the camera intrinsic parameter K. i Point cloud P i The process involves transforming the world coordinate system to the normalized device coordinate system (NDC), then rasterizing the transformed point cloud into a grid, and finally projecting the points in each grid onto the image pixels to obtain the rendered RGB-D image.

[0027] More specifically, the improved objective function L in step S23 local Specifically:

[0028]

[0029] Where mask(·) is the mask calculation for the rendered image; L local The first part represents color texture deviation, and the second part represents geometric depth deviation.

[0030] Furthermore, step S3 specifically includes:

[0031] S31. Calculate the global closed-loop cumulative pose. When the global view is closed and the pose estimation is correct, the image rendered from the point cloud after global cumulative pose transformation is considered to be consistent with the original image from the same view, and no global pose optimization is required; otherwise, global pose optimization is performed.

[0032] S32. Given a set of initially aligned point clouds and the optimized pose between adjacent viewpoints Minimize the globally accumulated pose The deviation between the rendered image and the original image from the transformed point cloud is used as the global optimization objective function to optimize the global pose.

[0033] More specifically, the global optimization objective function L in step S32 global Specifically:

[0034]

[0035] in, For the RGB image rendering process, In the depth image rendering process, the image rendering direction is opposite to the rendering direction in the global pose optimization process.

[0036] Based on the above technical solution, the present invention has at least the following beneficial effects:

[0037] 1. No real pose data required; This method constrains pose estimation through view rendering, eliminating the need for real pose data, significantly reducing the cost of dataset production, and reducing reliance on high-precision sensors and manual annotation, making it highly practical.

[0038] 2. Scene pose self-optimization: We abandon traditional feature learning methods for pose estimation and adopt a self-optimization approach for global pose prediction. This method is more adaptable to different scenes, improves the algorithm's generalization ability, and can handle scenes with different geometric structures, showing outstanding performance when facing complex or self-similar structures.

[0039] 3. Low cost and high efficiency training: This model is based solely on RGB-D data, and model training only requires predicting pose parameters. It does not require complex hardware auxiliary equipment, and the model can be trained and deployed in a low-cost environment, making it highly valuable for application. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of forward rendering and reverse rendering in the method proposed in this invention;

[0041] Figure 2 This is a framework diagram of a self-supervised 3D scene mapping method based on multi-view pose self-optimization proposed in this invention;

[0042] Figure 3 To render the specific flowchart;

[0043] Figure 4 A comparison chart of the path trajectories of four traditional methods;

[0044] Figure 5 A comparison diagram of the path trajectories of four supervised learning methods;

[0045] Figure 6 A comparison of point cloud errors after multi-view registration;

[0046] Figure 7 This is a comparison diagram of the path trajectories of the multi-view reconstruction method. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1-7 The present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0048] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0049] like Figure 2 As shown, this invention proposes a self-supervised 3D scene mapping method based on multi-view pose self-optimization (PSONet), which specifically includes the following steps:

[0050] S1. Given any adjacent data in the RGB-D closed-loop video stream, extract the corresponding two-dimensional matching points and map them to three-dimensional space to estimate the initial pose. Traditional methods such as neural radiation field (Nerf) correlation work have small viewing angle deviations in neighboring frames and have good relative initial positions to facilitate unknown pose estimation. However, this has high requirements for the viewing angle and time of the early data shooting. Therefore, for data with large viewing angle deviations in neighboring frames, this embodiment estimates the initial pose in the data preprocessing stage to reduce the viewing angle deviation.

[0051] In a preferred embodiment, step S1 specifically includes:

[0052] S11. Compared to unordered point clouds, RGB images can better display the detailed texture information of the target scene. Therefore, we estimate the initial pose between adjacent viewpoints by extracting more reliable image keypoints. Given RGB-D images from multiple viewpoints... Camera Insider Among them I i D represents the color information of the three channels. i Let N represent the number of input RGB-D images, and i represent the RGB-D image of the i-th frame (each frame corresponds to an image from a single viewpoint). For any two adjacent RGB images from viewpoints a and b, ORB features are extracted to obtain the 2D matching point {O}. 2a O 2b};

[0053] S12. It is worth noting that RGB images rely solely on scene texture information to match keypoints, lacking the geometric spatial constraints of the actual scene, leading to numerous mismatched points. The pose transformation of a rigid 3D point cloud should satisfy spatial compatibility (SC), meaning that any two points in space should maintain the same spatial distance after a rigid transformation. Therefore, correctly matched points should have the same spatial distance after being mapped to 3D space. This application uses a depth map aligned with the RGB image and the camera intrinsic parameter K. a ,K bMapping the matching points from adjacent viewpoints a and b to three-dimensional space yields the three-dimensional matching point {O}. 3a O 3b Then, based on spatial compatibility filtering, high-quality 3D matching points {H} are obtained. 3a H 3b The formula is expressed as:

[0054] {H 3a H 3b}=|d(a i ,a j )-d(b i ,b j )| <d th ;

[0055] Among them, (a i ,b i ) and (a j ,b j ) is {O 3a O 3b For any two pairs of 3D matching points in}, d(·,·) is the Euclidean distance, d th This is the distance threshold; ideally, |d(a) i ,a j )-d(b i ,b j The value of | should be 0, but due to the irregular distribution of point cloud data, the calculated value should be set to be less than a distance threshold d. th .

[0056] S13. Obtain high-quality three-dimensional matching points {H} 3a H 3b Then, the initial pose between adjacent viewpoints is estimated using the RANSAC pose estimator. Based on this, the data of each frame is initially aligned to reduce the deviation between adjacent viewpoints.

[0057] S2. Under the initial pose, map the RGB-D image to the three-dimensional space to obtain point cloud data; render the point cloud of any frame to the view of the adjacent frame to obtain the rendered RGB-D image, and minimize the deviation between the rendered RGB-D image and the original image of the adjacent view to optimize the inter-frame pose.

[0058] In a preferred embodiment, step S2 specifically includes:

[0059] S21. Given a set of RGB-D images under closed-loop continuous multi-view conditions. Camera internal parameters Where N is the number of input views, I i D represents the color information of the three channels. i Representing depth information, Ki Intrinsic parameters for each camera; using camera intrinsic parameters to convert RGB-D images (I i D i The point clouds P of each view are obtained by back-projection. i Specifically: First, a 3D mesh (3×H×W) is created. i Mapped to a 3D screen coordinate system, and then using the camera intrinsic parameter K. i The point cloud is obtained by inverse transformation of the screen coordinate system to the camera coordinate system, and then based on I... i and D i Pixel consistency maps RGB onto the point cloud, as expressed by the formula:

[0060] P i =p -1 (I i D i ,K i );

[0061] Where, p -1 (·,·,·) denotes the back projection function;

[0062] The work on S22 and Nerf has demonstrated that neural rendering can efficiently constrain scene pose estimation, and static scenes possess photometric and geometric consistency. Therefore, the rendered image at the correct pose viewpoint should be consistent with the original captured image. Similarly, if the pose estimation of adjacent viewpoints is correct, the rendered image of the point cloud from the neighboring viewpoints will also be consistent with the original image. Therefore, given a set of initially aligned point clouds (initial alignment is completed in step S1), First, the pose between adjacent viewpoints is optimized by minimizing the deviation between the rendered image and the original image of the point cloud from neighboring viewpoints as the local optimization objective function. The local optimization objective function is as follows:

[0063]

[0064] in, For the RGB image rendering process, This refers to the depth image rendering process; the rendering process specifically includes: Figure 3 As shown, using the camera intrinsic parameter K i Point cloud P i The process involves transforming the world coordinate system to the normalized device coordinate system (NDC), then performing mesh rasterization on the transformed point cloud, and finally projecting the points in each mesh raster onto the image pixels to obtain the rendered RGB-D image.

[0065] In addition, such as Figure 1 The diagrams showing forward and reverse rendering illustrate that, in this application, rendering from viewpoint a to viewpoint b is considered forward, and rendering from viewpoint b to viewpoint a is considered reverse; for example... Figure 2 As shown, This represents the rendered image from viewpoint a. This represents the rendering result from viewpoint a to viewpoint b.

[0066] S23. Due to spatial differences between camera viewpoints, some pixels are missing in the rendered image. Therefore, this application further calculates a mask for the rendered image during the rendering process, allowing unoccluded effective pixels to participate in the deviation calculation, thus improving the objective function. This alleviates the pixel loss between the rendered image and the original image caused by object occlusion due to viewpoint differences, which affects the pose constraint accuracy. The improved objective function L local Specifically:

[0067]

[0068] Where mask(·) is the mask calculation for the rendered image; L local The first part of the image represents color texture deviation, and the second part represents geometric depth deviation; thus, we can obtain the reliable pose between adjacent viewpoints. However, as the number of views increases, even slight pose deviations between adjacent viewpoints will lead to a significant shift in global accuracy. Therefore, global pose optimization is also particularly important.

[0069] Scene mapping requires stitching together single-view point clouds into a complete scene using poses from multiple adjacent views. Local optimization can constrain pose optimization of adjacent views, but global poses can also show significant deviations due to the accumulation of multiple views, resulting in low scene mapping accuracy. Therefore, step S3 is needed to perform global optimization at an appropriate time.

[0070] S3. Then, the point cloud of any frame is rendered backward along the camera path to the view of the adjacent frame to obtain the RGB-D image after backward rendering. The deviation between the image after backward rendering and the original image of the adjacent view is minimized to complete the global pose optimization.

[0071] In a preferred embodiment, step S3 specifically includes:

[0072] S31. Calculate the global closed-loop cumulative pose. When the global view is closed and the pose estimation is correct, the image rendered from the point cloud after global cumulative pose transformation is considered to be consistent with the original image from the same view, and no global pose optimization is required; otherwise, global pose optimization is performed.

[0073] S32. Given a set of initially aligned point clouds and the optimized pose between adjacent viewpoints To minimize the globally accumulated pose The deviation between the rendered image and the original image from the transformed point cloud is used as the global optimization objective function to optimize the global pose.

[0074] More specifically, the global optimization objective function L global Specifically:

[0075]

[0076] in, For the RGB image rendering process, In the depth image rendering process, the image rendering direction is opposite to the rendering direction in the global pose optimization process.

[0077] This concludes the description of the entire process of the method proposed in this invention. The specific flow of this method can also be referred to in Table 1 below:

[0078] Table 1. Detailed flowchart of the method proposed in this invention

[0079]

[0080]

[0081] Furthermore, in order to more reasonably analyze the optimization effect of the algorithm, this paper will delve into the comparison of point cloud registration effect and multi-view reconstruction effect, and compare it with other traditional point cloud registration algorithms and the most advanced SLAM algorithm to evaluate its performance and advantages in practical application scenarios.

[0082] The test dataset used in this experiment was provided free of charge by BlendSwap and covers six indoor scenes with different effects and spatial dimensions to evaluate the model's registration performance in different indoor scenes. Each scene dataset includes a corresponding RGB image and depth map, with device parameters including field of view and resolution. The camera perspective is a monocular structured light camera system, and the resolution of both the RGB images and depth maps is 640×480 pixels. The experiment aimed to ensure high continuity between adjacent frames, resulting in over 1000 viewpoints for each of the six different scenes in the dataset. PSONet is a self-supervised pose self-optimization registration algorithm; the model training process is essentially an optimization calculation process for unknown poses, eliminating the need for test and validation sets to verify model accuracy. In the experiment, downsampled viewpoint data from different scenes were input into PSONet for pose prediction, and the pose estimation errors were compared with other algorithms.

[0083] To test the registration performance of PSONet, this experiment compares it with traditional methods and supervised learning methods.

[0084] The registration performance of the traditional method is shown in the following experiment:

[0085] In the experiment, downsampled viewpoint data were input into PSONet for pose optimization iteration, and then the model-estimated inter-viewpoint pose transformation was output. To evaluate the registration performance of traditional methods and PSONet, Go-ICP, SAC-IA, and FGR were used to predict the poses between neighboring viewpoints after downsampling, and the prediction results were compared with the poses output by PSONet. Table 2 shows the performance comparison between PSONet and traditional registration methods on the proposed dataset. As can be seen from the table, PSONet has significant advantages in various metrics (i.e., relative rotation error RRE and relative translation error RTE) and exhibits good stability in six different indoor scenes.

[0086] Table 2 Comparison of registration performance using traditional methods

[0087]

[0088] Traditional methods typically use point-to-point distance loss to constrain the iterative optimization of pose, i.e., minimizing the average Euclidean distance between two point clouds. This method can directly measure the distance difference between point clouds, making the registered point clouds approximate the target shape. However, point-to-point distance loss is prone to getting trapped in local optima, causing the algorithm to fail to obtain accurate registration results. In contrast, PSONet predicts reliable initial poses based on 2D feature descriptors and spatial compatibility, and uses viewpoint rendering to more effectively constrain poses. Furthermore, global pose optimization in the model further improves the accuracy of pose estimation.

[0089] In multi-view registration tasks, the path trajectory estimation error T abs Global pose estimation is an important metric for evaluating the performance of different registration methods. It describes the spatial distance difference between the estimated path and the real path and can be used to compare the accuracy and stability of global pose prediction using different methods. In the experiments, Go-ICP, SAC-IA, and FGR algorithms were used to estimate the pose between downsampled viewpoints and synthesize the estimated path trajectories. Then, the downsampled viewpoint data was input into PSONet to obtain global pose estimation, and the predicted path trajectories were calculated. Finally, the path trajectories estimated by the above methods were compared with the real paths in the dataset. The registration performance of Go-ICP, SAC-IA, and FGR algorithms was relatively low, showing significant differences from the real path trajectory starting from the second segment of the estimated path trajectory. This section visualizes the predicted path trajectory and the real path trajectory, such as... Figure 4 As shown, the path trajectory estimated by PSONet proposed in this invention is consistent with the actual path trajectory, demonstrating the stability and accuracy of the PSONet algorithm. Further analysis reveals that multi-view registration tasks require algorithms with high stability and accuracy; pose errors between adjacent frames will cause global error accumulation, leading to a decrease in the accuracy of multi-view registration.

[0090] Next, we will conduct performance experiments on the configuration of supervised learning methods:

[0091] Predator, D3Feat, and GCNet are currently high-performance pairwise point cloud registration algorithms. They calculate corresponding points and predict the transformation pose matrix by constraining the feature representation of the network through the real pose. In the experiments, feature extractors for Predator, D3Feat, and GCNet were trained using pre-downsampled viewpoint data. To adapt the algorithms to point cloud registration with low overlap, the viewpoint interval between the pairwise registered point clouds was increased in the experiments, thereby reducing the overlap of the point clouds to be registered. Finally, the downsampled viewpoint data was input into Predator, D3Feat, and GCNet respectively to predict the pose between neighboring viewpoints after downsampling, and compared with the pose predicted by PSONet. The performance comparison results are shown in Table 3 below:

[0092] Table 3 Comparison of Registration Performance of Supervised Learning Methods

[0093]

[0094] As shown in Table 3, compared to the aforementioned supervised algorithms, PSONet demonstrates a significant advantage in registration metrics across six different indoor scenes. Predator and D3Feat both employ KPCov-based algorithms.

[32] The U-Net network architecture is used to represent multi-scale features of the network model through encoders and decoders. As shown in Table 3, PSONet's RRE and RTE are significantly better than the supervised learning algorithms mentioned above, indicating that the algorithm has high accuracy and robustness.

[0095] To better represent the geometric features of point clouds, Predator, D3Feat, and GCNet employ KPCov or graph convolution strategies to directly manipulate or learn the information representation between points. However, compared to other convolution methods, KPCov and graph convolution increase the computational requirements of the model to some extent, typically requiring several hours to complete model training. In contrast, PSONet abandons feature learning and directly estimates the pose through global pose self-optimization, which greatly reduces the number of model parameters and thus improves the training efficiency. Therefore, PSONet can achieve pose optimization estimation in a short time without training with a feature extractor or constraints from the true pose.

[0096] Most existing point cloud registration algorithms focus on pairwise point cloud registration tasks, registering each pair of point clouds separately. This means that the registration result of each pair is affected by the previous registration results. As the number of views to be registered increases, the accumulated error makes the point cloud registration result inaccurate. Figure 5As shown, the Predator, D3Feat, and GCNet point cloud registration algorithms have relatively small errors in the early stages, but as the number of viewpoints increases, the path trajectories gradually show significant errors. In contrast, PSONet employs a global pose optimization strategy, ensuring that the estimated path trajectory maintains high accuracy throughout, without exhibiting significant errors.

[0097] Furthermore, many advanced SLAM-related works based on RGB and depth images have emerged in the existing technology, such as ORB-SLAM2, ORB-SLAM3, and NICE-SLAM. These works include pose estimation tasks based on RGB and depth images, and their loop closure detection strategies can effectively eliminate accumulated errors, enabling these works to exhibit excellent performance in multi-view reconstruction tasks. To better test the multi-view registration performance of PSONet, this experiment compares its pose prediction results with state-of-the-art SLAM algorithms, as shown in Table 4.

[0098] Table 4. Performance Comparison of Multi-View Reconstruction

[0099]

[0100]

[0101] ORB-SLAM2, ORB-SLAM3, and NICE-SLAM have strict requirements for data continuity, demanding a sufficiently high degree of overlap between adjacent frame images and point clouds. Insufficient overlap between adjacent frames will lead to incorrect motion pose estimation, resulting in significant deviations in 3D mapping. NICE-SLAM introduces implicit neural representations into the SLAM task, thus addressing the memory consumption issue of large-scene point clouds. However, implicit neural representations cannot constrain pose estimation using accurate explicit geometric point clouds, leading to decreased accuracy in pose estimation. In contrast, ORB-SLAM2 and ORB-SLAM3 can directly constrain pose estimation using explicit point clouds, resulting in better reconstruction performance.

[0102] To visually demonstrate the point cloud error after multi-view registration, the experiment used the predicted poses of ORB-SLAM2, ORB-SLAM3, NICE-SLAM, and PSONet to stitch together multi-view point clouds, and calculated the distance error between the stitched point cloud and the original complete point cloud. For example... Figure 6As shown, the error fluctuation range of PSONet and ORB-SLAM3 is 0-0.0007m, that of NICE-SLAM is 0-0.003m, and that of ORB-SLAM2 is 0-0.007m. The data indicates that PSONet's registration performance reaches or even surpasses that of the state-of-the-art ORB-SLAM3. Its multi-view point cloud stitching error is significantly lower than that of ORB-SLAM2 and NICE-SLAM, demonstrating high accuracy. Furthermore, PSONet's point cloud error distribution is more uniform, exhibiting excellent stability.

[0103] However, existing SLAM methods have strict requirements for data continuity and cannot solve the registration problem of point clouds with low overlap. Furthermore, redundant data processing causes ORB-SLAM2, ORB-SLAM3, and NICE-SLAM to consume a considerable amount of time to estimate camera pose. NICE-SLAM, in particular, employs a joint optimization approach of implicit neural modeling and pose estimation, typically requiring several hours to estimate camera pose and model a complete 3D scene. PSONet, on the other hand, effectively solves the registration problem caused by low-overlap data, and its minimal network parameters allow the algorithm to achieve optimized global pose estimation in a short time.

[0104] like Figure 7 As shown, the path trajectories estimated by ORB-SLAM2, ORB-SLAM3, NICE-SLAM, and PSONet are consistent with the real paths, and the cumulative error problem of pairwise point cloud registration algorithms is not observed. Experiments show that the global pose optimization strategy in PSONet can effectively eliminate cumulative errors and achieve accurate multi-view point cloud registration.

[0105] In summary, all the experiments demonstrate that the proposed PSONet method performs exceptionally well in multi-view point cloud registration and reconstruction tasks, particularly in handling complex indoor scenes and low-overlap data. Compared to traditional point cloud registration methods, PSONet achieves significant progress in both the accuracy and robustness of pose estimation, especially in path trajectory accuracy and point cloud error uniformity. Through global pose optimization, PSONet successfully avoids the cumulative error problem common in traditional methods and significantly improves the accuracy and stability of point cloud reconstruction.

[0106] Furthermore, this method demonstrates superior computational efficiency. Compared to some advanced SLAM algorithms (such as ORB-SLAM2, ORB-SLAM3, and NICE-SLAM), PSONet can process low-overlap point cloud data more efficiently and achieve global pose optimization estimation at a lower computational cost. This characteristic makes PSONet a strong advantage in practical applications, especially in environments with limited computational resources.

[0107] In summary, PSONet not only advances point cloud registration technology but also provides an efficient and low-cost solution capable of achieving high-precision multi-view point cloud registration and reconstruction without relying on complex hardware or real pose data. Future work can build upon this foundation to further optimize algorithm performance, expand its application in larger-scale scenarios, explore the possibility of fusing with other sensor data, and enhance its adaptability and generalization capabilities in practical applications.

[0108] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0109] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A self-supervised 3D scene mapping method based on multi-view pose self-optimization, characterized in that, Specifically, the following steps are included: S1. Given any adjacent data in an RGB-D closed-loop video stream, extract the corresponding two-dimensional matching points and map them to three-dimensional space to estimate the initial pose. S2. Under the initial pose, the RGB-D image is mapped to 3D space to obtain point cloud data; the point cloud of any frame is rendered to the viewpoint of the adjacent frame to obtain the rendered RGB-D image, and the deviation between the rendered RGB-D image and the original image of the adjacent viewpoint is minimized to optimize the inter-frame pose; the optimization of the inter-frame pose specifically involves: Given a set of initially aligned point clouds , The number of input RGB-D images is represented by , and i represents the input RGB-D image of the i-th frame. The pose between adjacent viewpoints is optimized by minimizing the deviation between the rendered image and the original image of the point cloud of neighboring viewpoints as the local optimization objective function. ; In the rendering process, a mask for the rendered image is calculated, allowing unmasked effective pixels to participate in the deviation calculation, thus improving the objective function. Specifically: ； in, For the RGB image rendering process, For the depth image rendering process; Calculate the mask for rendering the image; This represents the color information of the three channels. Represents depth information; The first part represents color texture deviation, and the second part represents geometric depth deviation; S3. Then, the point cloud of any frame is sequentially rendered backwards along the camera path to the viewpoint of the adjacent frame to obtain the inverted RGB-D image. The deviation between the inverted image and the original image of the adjacent viewpoint is minimized to complete the global pose optimization; the global pose optimization specifically includes: Given a set of initially aligned point clouds and the optimized pose between adjacent viewpoints To minimize the globally accumulated pose The deviation between the rendered image and the original image from the transformed point cloud is used as the global optimization objective function to optimize the global pose; the global optimization objective function... Specifically: ； In global pose optimization, the image rendering direction is opposite to that in local pose optimization.

2. The self-supervised 3D scene mapping method based on multi-view pose self-optimization according to claim 1, characterized in that, Step S1 specifically includes: S11. Given RGB-D images from multiple perspectives Camera internal parameters For any two adjacent RGB images from viewpoints a and b, ORB features are extracted to obtain 2D matching points. ; S12, Use depth maps and camera intrinsics aligned with RGB images. , Mapping the 2D matching points from adjacent viewpoints a and b to 3D space yields 3D matching points. Then, high-quality 3D matching points are obtained through spatial compatibility screening. The formula is expressed as: ； in, and for Any two pairs of 3D matching points, For Euclidean distance, Distance threshold; S13. Obtain high-quality 3D matching points. Then, the initial pose between adjacent viewpoints is estimated using the pose estimator RANSAC.

3. The self-supervised 3D scene mapping method based on multi-view pose self-optimization according to claim 1, characterized in that, In step S2, the step of mapping the RGB-D image to three-dimensional space to obtain point cloud data under the initial pose specifically involves: Given a set of closed-loop continuous multi-view RGB-D images Camera internal parameters , Intrinsic parameters for each camera; using camera intrinsic parameters to convert RGB-D images Point clouds of each view are obtained by back-projection. Specifically: First, a 3D mesh is created. Will Mapped to a 3D screen coordinate system, then using camera intrinsics. The point cloud is obtained by inversely transforming the screen coordinate system to the camera coordinate system, and then... and Pixel consistency maps RGB onto the point cloud, as expressed by the formula: ； in, This represents the back projection function.

4. The self-supervised 3D scene mapping method based on multi-view pose self-optimization according to claim 1, characterized in that, In step S2, the rendering process specifically involves: utilizing camera intrinsic parameters Point cloud The process involves transforming the world coordinate system to the normalized device coordinate system (NDC), then rasterizing the transformed point cloud into a grid, and finally projecting the points in each grid onto the image pixels to obtain the rendered RGB-D image.

5. The self-supervised 3D scene mapping method based on multi-view pose self-optimization according to claim 1, characterized in that, Step S3 also includes: Calculate global closed-loop cumulative pose When the global view is closed and the pose estimation is correct, the image rendered by the point cloud after global cumulative pose transformation is considered to be consistent with the original image at the same view, and no global pose optimization is required; otherwise, global pose optimization is performed.

Images