Binocular camera high-precision pose evaluation method based on 3DGS
By employing a high-precision pose estimation method based on 3DGS binocular cameras and utilizing deep learning and 3D Gaussian point optimization techniques, the problem of insufficient pose estimation accuracy of binocular cameras in complex scenes is solved, achieving high-precision and robust pose estimation.
Patent Information
- Application Number
- CN202511560572.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-29
- Publication Date
- 2026-02-17
AI Technical Summary
In existing technologies, binocular cameras suffer from insufficient accuracy and poor robustness in high-precision pose estimation, making it difficult to meet the needs of complex scenarios.
A high-precision pose evaluation method based on 3DGS binocular cameras is adopted. Features are extracted through deep learning, horizontal parallax is calculated, dense color point cloud is generated, and 3D Gaussian points are used to optimize camera pose. The 3D Gaussian model is iteratively optimized to improve accuracy.
It achieves high-precision and robust pose estimation, adapts to complex environments, and outputs high-precision camera pose.
Smart Images
Figure CN121544702A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of visual image data processing technology, and specifically relates to a high-precision pose evaluation method for binocular cameras based on 3DGS, which is suitable for scenarios that accurately acquire camera position and pose information. Background Technology
[0002] Pose perception is one of the core technologies in computer vision, and its accuracy directly affects the performance of subsequent applications. Currently, mainstream pose perception solutions rely on sensors such as LiDAR, inertial measurement units (IMUs), RGBD cameras, or binocular cameras.
[0003] LiDAR estimates depth by actively emitting laser beams and receiving echoes. It is not sensitive to sunlight, but it cannot collect environmental color information, and the equipment is expensive and consumes a lot of power.
[0004] RGBD cameras combine depth and color information to output colored point clouds, but due to limitations in active light emission technology, they cannot adapt to strong light outdoor environments, and the depth acquisition distance is usually only a few meters.
[0005] IMUs have high frame rates and excellent short-term accuracy, but they are prone to severe drift during long-term operation, leading to data distortion.
[0006] Passive binocular cameras are simple in structure, low in cost, and have no active light-emitting module. They recover depth through the principle of parallax, and theoretically have a long detection range and strong environmental adaptability. However, their traditional depth recovery methods have problems such as insufficient accuracy and poor robustness to complex scenes, making it difficult to meet the requirements of high-precision pose estimation. Summary of the Invention
[0007] This invention provides a high-precision pose estimation method for binocular cameras based on 3DGS, which addresses the technical problem of how to leverage the advantages of binocular cameras and combine them with advanced 3D modeling and optimization techniques to improve the accuracy of pose estimation. The method adds new Gaussian points to the 3DGS by calculating new point clouds, iteratively optimizes a set of 3DGS with similar poses, optimizes the camera pose, and outputs a high-precision camera pose.
[0008] To achieve the above objectives, the present invention is implemented through the following technical solution:
[0009] A high-precision pose evaluation method for a stereo camera based on 3DGS includes the following steps:
[0010] Step 1: Acquire the left and right images captured by the stereo camera, and correct the input stereo images;
[0011] Step 2: Use deep learning methods to extract features from the corrected left and right images to estimate the depth, and then extract features from the left and right depth maps after monocular depth estimation to achieve horizontal disparity estimation.
[0012] Step 3: Perform calculations on the obtained horizontal parallax and the intrinsic parameter matrix of the binocular camera to calculate the depth information of the environment and output a dense environmental color point cloud at the pixel density level.
[0013] Step 4: Using the camera pose of the initial frame as the zero position, the obtained dense color point cloud of the environment is used to initialize the 3D Gaussian representation of the environment. The 3D Gaussian points are used as the environment representation units. By optimizing the error between the 3D Gaussian model or 3DGS and the camera-acquired image, the 3D Gaussian points used for environment representation are iteratively optimized to maintain the global color point cloud.
[0014] Step 5: For subsequent frames, match the dense color point cloud of the current frame with the maintained global color point cloud to obtain the preliminary camera pose.
[0015] Step 6: Add the initial pose of the new frame camera and the corresponding image to the 3DGS optimization queue, and register the newly acquired point cloud to the global color point cloud;
[0016] Step 7: Add new Gaussian points to 3DGS by calculating the feature information of the newly added point cloud;
[0017] Step 8: Iteratively optimize a set of 3D Gaussian models with similar poses, simultaneously optimize the 3D Gaussian point parameters of the environment and the camera pose parameters, and output high-precision camera pose.
[0018] Optionally, in step 1, the correction of the binocular images is based on the intrinsic and extrinsic parameters of the binocular camera, and the correction includes eliminating image distortion and making the epipolar lines of the left and right images parallel.
[0019] Optionally, in step 2, the deep learning method used is a convolutional neural network model. The convolutional neural network model learns the mapping relationship between image features and depth and disparity through training, thereby improving the robustness and accuracy of disparity estimation.
[0020] Optionally, in step 4, the objective function for optimizing the 3D Gaussian model and the image acquisition error by the camera is a weighted sum of pixel-level color and depth errors, and the 3D Gaussian point parameters are iteratively updated using a gradient descent algorithm.
[0021] Optionally, in step 5, the matching of dense color point cloud and global color point cloud adopts a matching algorithm based on feature point descriptors. The matching of corresponding points is achieved by calculating the feature similarity between two point clouds, and then the preliminary camera pose is solved.
[0022] Optionally, in step 6, the point cloud registration uses the Iterative Nearest Point or ICP algorithm to perform fine registration between the newly acquired point cloud and the global color point cloud, ensuring the coordinate consistency between the new point cloud and the global point cloud.
[0023] Optionally, in step 7, the conditions for adding new Gaussian points are that the overlap between the new point cloud region and the existing 3D Gaussian point coverage region is less than a preset threshold, and the feature entropy value of the new point cloud is higher than a set threshold.
[0024] Optionally, in step 8, the number of 3D Gaussian models for iterative optimization is determined by a sliding window mechanism. The window contains the 3D Gaussian models corresponding to the current frame and the previous N frames, and the value of N ranges from 3 to 10.
[0025] Optionally, in step 8, the optimization objective of the camera pose parameters is to minimize the error between the pixel coordinates of the 3D Gaussian point projected onto the image plane and the actual image pixel coordinates.
[0026] Optionally, the method may also include a scalability adaptation step:
[0027] When the data source is replaced with a monocular camera, the number of 3DGS iterations is increased to compensate for the decrease in initial pose estimation accuracy caused by the lack of parallax information.
[0028] When the data source is replaced with an infrared camera, the parameters of the feature extraction algorithm are adjusted to adapt to the grayscale features of the infrared image.
[0029] The beneficial effects of this invention are:
[0030] This invention uses a binocular camera as the data source and leverages the camera's extrinsic characteristics for high-precision depth estimation. Specifically, deep learning methods are used to extract features from the left and right images to estimate depth. Features are then extracted from the left and right depth maps obtained through monocular depth estimation to estimate horizontal disparity. The resulting horizontal disparity is used to calculate the environmental depth information through operations with the intrinsic parameter matrix. The resulting depth map exhibits good environmental adaptability and robustness, and can output dense environmental color point clouds at the pixel density level. The initial image frame uses the dense environmental color point cloud estimated by the binocular camera to initialize the 3D Gaussian representation of the environment, using 3D Gaussian points as the representation units of the environment. By optimizing the error between 3DGS and the image acquired by the camera, the 3D Gaussian points used to represent the environment can be continuously optimized, and the global color point cloud can be maintained. For a new image frame, the dense color point cloud of the environment within the binocular acquisition range is estimated first. The obtained dense color point cloud is matched with the global color point cloud representing the environment to obtain the preliminary camera pose. The initial pose and image of the camera in the new frame are added to the 3DGS optimization queue, and the newly acquired point cloud is registered to the global color point cloud. Finally, new Gaussian points are added to the 3DGS by calculating the newly added point cloud. A set of 3DGS with similar poses is iteratively optimized to optimize the camera pose and output a high-precision camera pose. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 This is a schematic diagram of the workflow of the present invention;
[0033] Figure 2 This is a color point cloud image of a dense environment according to the present invention;
[0034] Figure 3 This is the global color point cloud map of the present invention. Detailed Implementation
[0035] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0036] Example 1;
[0037] The hardware used in this invention includes: a binocular camera (model: Basler acA2500-14uc, baseline length 120mm, resolution 2592×1944), and an industrial computer (CPU: Intel Core i7-12700H, GPU: NVIDIA RTX3070Ti, memory 32GB); the software environment includes: operating system Ubuntu 20.04 LTS, deep learning framework PyTorch 1.12.1, point cloud processing library PCL 1.12, and image processing library OpenCV 4.5.5.
[0038] Specifically, such as Figure 1 As shown, this embodiment provides a high-precision pose evaluation method for a binocular camera based on 3DGS, including the following steps:
[0039] Step 1: Acquire the left and right images captured by the stereo camera, and correct the input stereo images;
[0040] Step 2: Use deep learning methods to extract features from the corrected left and right images to estimate the depth, and then extract features from the left and right depth maps after monocular depth estimation to achieve horizontal disparity estimation.
[0041] Step 3: Perform calculations using the obtained horizontal parallax and the intrinsic parameter matrix of the stereo camera to determine the depth information of the environment, outputting pixel density level data such as... Figure 2 The dense environment shown is a colored point cloud;
[0042] Step 4: Using the initial frame camera pose as the zero position, initialize the 3D Gaussian representation of the environment using the obtained dense color point cloud. Use these 3D Gaussian points as the environment representation unit. Iteratively optimize the 3D Gaussian points used for environment representation by optimizing the error between the 3D Gaussian model or 3DGS and the camera-acquired image, maintaining the relationship as follows: Figure 3 The global color point cloud shown; by using deep learning methods, binocular depth estimation can obtain high-precision and robust environmental dense color point clouds and global color point clouds, resulting in a binocular estimated color point cloud;
[0043] Step 5: For subsequent frames, match the dense color point cloud of the current frame with the maintained global color point cloud to obtain the preliminary camera pose.
[0044] Step 6: Add the initial pose of the new frame camera and the corresponding image to the 3DGS optimization queue, and register the newly acquired point cloud to the global color point cloud;
[0045] Step 7: Add new Gaussian points to 3DGS by calculating the feature information of the newly added point cloud;
[0046] Step 8: Iteratively optimize a set of 3D Gaussian models with similar poses, simultaneously optimize the 3D Gaussian point parameters of the environment and the camera pose parameters, and output high-precision camera pose.
[0047] In step 1, the calibration of the stereo images is based on the intrinsic and extrinsic parameters of the stereo camera. The calibration includes eliminating image distortion and ensuring the epipolar lines of the left and right images are parallel. Based on the intrinsic parameters (focal length, principal point coordinates, distortion coefficients) and extrinsic parameters (rotation matrix, translation vector) obtained from the stereo camera calibration, an epipolar calibration algorithm (such as the Bouguet algorithm) is used to map the left and right images onto a coplanar calibration plane with parallel epipolar lines, ensuring that corresponding pixels only have horizontal parallax. After calibration, the accuracy needs to be verified through reprojection error. When the average reprojection error exceeds 1 pixel, recalibration or parameter optimization is performed.
[0048] In step 2, the deep learning method employed is a convolutional neural network (CNN) model. The CNN model learns the mapping relationship between image features and depth and disparity through training, improving the robustness and accuracy of disparity estimation. The CNN model can be specifically defined as a Transformer-based binocular disparity estimation model (e.g., DPT, Monodepth3), which captures long-distance pixel associations through multi-scale feature fusion and self-attention mechanisms, improving the disparity estimation accuracy in weakly textured regions (e.g., walls, sky). Simultaneously, model training needs to include data augmentation steps (e.g., random cropping, brightness adjustment, noise addition) to enhance adaptability to complex environments.
[0049] In step 3, the depth information of the environment is calculated using the conversion relationship between horizontal parallax and depth: ;in, For depth value, The focal length of the camera (in pixels). The binocular baseline distance. To address the horizontal parallax (the difference in pixel column coordinates between the left and right images), the calculated depth map needs to undergo bilateral filtering or guided filtering to denoise the image, preserving edge information while reducing noise interference.
[0050] In step 4, the objective function for optimizing the errors between the 3D Gaussian model and the camera-acquired images is a weighted sum of pixel-level color and depth errors. The gradient descent algorithm is used to iteratively update the 3D Gaussian point parameters. The parameters of the 3D Gaussian points should include: spatial coordinates. Covariance matrix (Describe the spatial distribution range), color attributes and opacity During initialization, the efficiency can be improved by clustering dense point clouds (e.g., using the DBSCAN algorithm), with the cluster centers used as initial Gaussian centers and the cluster variance used as initial covariance.
[0051] In addition to color errors (e.g., mean square error of RGB channels) and depth errors (e.g., L1 loss between predicted depth and point cloud depth), regularization terms can be added to the objective function: such as sparsity regularization of Gaussian point opacity (suppressing redundant Gaussian points) and trace constraint of covariance matrix (avoiding excessive stretching of Gaussian distribution) to prevent model overfitting.
[0052] In step 5, the matching of dense color point clouds and global color point clouds employs a feature point descriptor-based matching algorithm. This algorithm calculates the feature similarity between two point clouds to achieve corresponding point matching, thereby solving for the initial camera pose. Feature matching based on point clouds uses either FPFH (Fast Point Feature Histogram) or SHOT (Signature of Histograms of Orientations) descriptors. The former is suitable for fast matching, while the latter performs better in rotation invariance. After matching, mismatched points are eliminated using the RANSAC (Random Sample Consensus) algorithm, and the PnP (Perspective-n-Point) algorithm is then used to solve for the initial pose, ensuring that the initial pose error is controlled within 5° (rotation) and 0.5m (translation).
[0053] In step 6, point cloud registration employs an iterative nearest point (INP) or ICP algorithm to precisely register the newly acquired point cloud with the global color point cloud, ensuring coordinate consistency between the new and global point clouds. An improved ICP algorithm (e.g., Point-to-Plane ICP) is used to enhance registration accuracy by minimizing the distance error from a point to its corresponding plane, making it particularly suitable for indoor scenes with numerous planes. After registration, the root mean square error (RMSE) of the overlapping point cloud region needs to be calculated. If the RMSE exceeds a preset threshold (e.g., 0.05m), the process returns to step 5 for feature matching to be performed again.
[0054] In step 7, the conditions for adding new Gaussian points are that the overlap between the new point cloud region and the existing 3D Gaussian point coverage area is less than a preset threshold, and the feature entropy value of the new point cloud is higher than a set threshold. The feature entropy is calculated based on the normal vector distribution, color histogram, or local curvature of the point cloud: for a new point cloud region, if its normal vector direction entropy (reflecting surface complexity) is higher than a threshold (e.g., 0.8), or its color histogram entropy (reflecting texture richness) is higher than a threshold (e.g., 1.2), it is determined to be a key region where Gaussian points need to be added to improve the expression accuracy of complex scenes.
[0055] In step 8, the number of 3D Gaussian models for iterative optimization is determined using a sliding window mechanism. The window contains the 3D Gaussian models corresponding to the current frame and the previous N frames, with N ranging from 3 to 10. The window size N can be dynamically adjusted according to the scene: in fast-moving scenes (e.g., camera translation speed > 1 m / s), a smaller N (3-5) is used to reduce computational latency; in static or slow-moving scenes, a larger N (6-10) is used to utilize more historical frame information to improve pose stability. The optimization objective of the camera pose parameters is to minimize the error between the pixel coordinates of the 3D Gaussian points projected onto the image plane and the actual image pixel coordinates.
[0056] In addition, the method in this embodiment also includes an extensibility adaptation step:
[0057] When the data source is replaced with a monocular camera, the number of 3DGS iterations is increased to compensate for the decrease in initial pose estimation accuracy caused by the lack of parallax information.
[0058] When the data source is replaced with an infrared camera, the parameters of the feature extraction algorithm are adjusted to adapt to the grayscale features of the infrared image.
[0059] By adjusting parameters, compatibility with two common alternative data sources, monocular cameras and infrared cameras, is achieved. This retains the core framework of the method (3DGS optimization, point cloud matching, and pose iteration optimization) while addressing the inherent defects of different data sources and broadening the applicable scenarios of the technology.
[0060] To address the characteristics of infrared images, such as uniform grayscale and sparse texture, the parameters of the convolutional neural network or feature extraction module are optimized to enhance the ability to capture weak textures and grayscale gradient features.
[0061] Example 2;
[0062] Based on Embodiment 1, the camera of the present invention can be extended to all cameras, including: monocular cameras and infrared cameras. For example, when using a monocular camera, in step 2, due to the lack of contrast information, disparity estimation cannot be performed, and the depth estimation performance will decrease to some extent. As a result, the initial pose estimation accuracy will decrease to some extent, requiring more iterations in the subsequent frame 3DGS iteration optimization process.
[0063] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope described in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A high-precision pose evaluation method based on a 3DGS binocular camera, characterized in that, The method comprises the following steps: Step 1: obtaining left and right images collected by a binocular camera, and correcting the input binocular image; Step 2: using a deep learning method to respectively extract features of the corrected left and right images to estimate depth, and then extracting features of the left and right depth maps estimated by monocular depth estimation to realize horizontal disparity estimation; Step 3: performing operation on the obtained horizontal disparity and an internal parameter matrix of the binocular camera to calculate depth information of the environment, and outputting pixel density level dense environment color point cloud; Step 4: taking an initial frame camera pose as zero position, using the obtained dense color point cloud to initialize 3D Gaussian representation of the environment, taking 3D Gaussian points as environment expression units, and iteratively optimizing the 3D Gaussian points used for environment expression by optimizing errors of the 3D Gaussian model or the 3DGS and the camera collected image, and maintaining the global color point cloud; Step 5: for subsequent frames, matching the dense color point cloud of the current frame with the maintained global color point cloud to obtain a preliminary camera pose; Step 6: adding the initial pose of the new frame camera and the corresponding image to the 3DGS optimization queue, and registering the newly collected point cloud to the global color point cloud; Step 7: adding new Gaussian points to the 3DGS by calculating feature information of the newly added point cloud; Step 8: iteratively optimizing a group of 3D Gaussian models with similar poses, synchronously optimizing 3D Gaussian point parameters and camera pose parameters of the environment, and outputting high-precision camera pose.
2. The high-precision pose evaluation method for binocular camera based on 3DGS according to claim 1, characterized in that, In the step 1, the correction of the binocular image is based on internal and external parameter data of the binocular camera, and the correction content includes eliminating image distortion and making the epipolar lines of the left and right images parallel.
3. The high-precision pose evaluation method for binocular camera based on 3DGS according to claim 1, characterized in that, In the step 2, the deep learning method used is a convolutional neural network model, which learns the mapping relationship between image features and depth and disparity through training, thereby improving the robustness and accuracy of disparity estimation.
4. The high-precision pose evaluation method for binocular camera based on 3DGS according to claim 1, characterized in that, In the step 4, the objective function of the optimization of the errors of the 3D Gaussian model and the camera collected image is a weighted sum of pixel level color error and depth error, and the iterative update of the 3D Gaussian point parameters is realized by a gradient descent algorithm.
5. The high-precision pose evaluation method for binocular camera based on 3DGS according to claim 1, characterized in that, In the step 5, the matching of the dense color point cloud and the global color point cloud uses a matching algorithm based on feature point descriptors, and the corresponding point matching is realized by calculating the feature similarity between the two point clouds, and then the preliminary camera pose is solved.
6. The high-precision pose evaluation method for binocular camera based on 3DGS according to claim 1, characterized in that, In the step 6, the point cloud registration uses an iterative closest point or ICP algorithm to accurately register the newly collected point cloud with the global color point cloud, thereby ensuring the coordinate consistency of the new point cloud and the global point cloud.
7. The high-precision pose evaluation method for binocular camera based on 3DGS according to claim 1, characterized in that, In the step 7, the adding condition of the new Gaussian point is that the overlap degree of the new point cloud region and the existing 3D Gaussian point coverage region is lower than a preset threshold, and the feature entropy value of the new point cloud is higher than a set threshold.
8. The high-precision pose evaluation method for binocular camera based on 3DGS according to claim 1, characterized in that, In the step 8, the number of iteratively optimized 3D Gaussian models is determined by a sliding window mechanism, and the window contains the 3D Gaussian models corresponding to the current frame and the previous N frames, and the value range of N is 3-10.
9. The high-precision pose evaluation method for binocular camera based on 3D GS according to claim 1, characterized in that, In the step 8, the optimization target of the camera pose parameter is to minimize the error between the pixel coordinates of the 3D Gaussian point projected to the image plane and the actual image pixel coordinates.
10. The high-precision pose evaluation method of binocular camera based on 3DGS according to any one of claims 1-9, characterized in that, The method further comprises a scalability adaptation step: When the data source is replaced by a monocular camera, the number of 3DGS iterative optimization is increased to compensate for the decrease in the accuracy of the initial pose estimation due to the lack of parallax information. When the data source is replaced by an infrared camera, the parameters of the feature extraction algorithm are adjusted to adapt to the gray scale features of the infrared image.