Monocular vision-based map construction method, system and equipment

Abstract

The invention relates to a monocular vision-based map construction method, system and equipment. The monocular vision-based map construction method comprises the following steps of: acquiring a current frame image, a key frame image and a sparse point cloud map; extracting image information of the current frame image, and determining a first camera attitude; determining a prior depth map of the current frame image on the key frame image; constructing a first Gaussian model according to the first camera attitude, the image information, the sparse point cloud map, the prior depth map and the current frame image, and optimizing the first camera attitude to obtain a second camera attitude; wherein the Gaussian model comprises Gaussian distribution of a plurality of scenes; and obtaining a second Gaussian model according to the sparse point cloud map, the prior depth map, the image information and the second camera attitude, and adding the second Gaussian model to the Gaussian map according to the second camera attitude. The method can improve the positioning and map building precision, and is applied to the technical field of computer vision and robots.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and robotics technology, and in particular to a monocular vision-based map construction method, system and device. Background Art

[0002] Monocular vision-based simultaneous localization and mapping (SLAM) holds broad application prospects in areas such as mobile robotics, augmented reality, and autonomous driving. However, existing monocular vision SLAM systems have limitations in their map representation methods. For example, using point cloud maps or grid maps can lead to information loss or excessive storage requirements when representing complex scenes, making it difficult to meet real-time and high-precision requirements. Furthermore, inaccurate depth estimation leads to insufficient positioning and mapping accuracy. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a monocular vision-based map construction method, system and device, which can improve the accuracy of positioning and map construction.

[0004] In one aspect, the present invention provides a method for constructing a map based on monocular vision, comprising the following steps:

[0005] Get the current frame image, key frame image and sparse point cloud map;

[0006] Extracting image information of the current frame image and determining a first camera posture;

[0007] Determining a priori depth map of the current frame image on the key frame image; constructing a first Gaussian model based on the first camera pose, the image information, the sparse point cloud map, the priori depth map, and the current frame image, and optimizing the first camera pose to obtain a second camera pose; wherein the Gaussian model includes Gaussian distributions of multiple scenes;

[0008] A second Gaussian model is obtained according to the sparse point cloud map, the prior depth map, the image information, and the second camera pose, and the second Gaussian model is added to the Gaussian map according to the second camera pose.

[0009] Optionally, extracting image information of the current frame image and determining a first camera posture specifically includes:

[0010] Divide the current frame image into several sub-images based on a deep learning algorithm, extract initial information, and determine a third camera posture based on the initial information;

[0011] An inter-block optical flow and a reprojection error of the sub-image are calculated, and the inter-block optical flow and the reprojection error are processed using a nonlinear optimization method, and the third camera pose and the initial information are updated to obtain the image information and the first camera pose.

[0012] Optionally, determining a third camera posture according to the image feature specifically includes:

[0013] If the current frame image is a frame image before the Nth frame or the Nth frame image, obtain initial information of several frame images next to the current frame, and calculate a basic matrix based on the initial information of the current frame image and the image information of the several frame images next to the current frame; calculate an essential matrix based on the basic matrix and the intrinsic parameter matrix of the camera, decompose the essential matrix to obtain the third camera pose; wherein N is a natural number greater than or equal to 1; if the current frame image is a frame image after the Nth frame, obtain the motion speed and time interval of the current frame image and the previous frame image, and determine the third camera pose based on the motion speed, the time interval, and the camera pose of the previous frame image.

[0014] Optionally, determining the prior depth map of the current frame image on the key frame image specifically includes:

[0015] Extracting multi-scale features of the current frame image;

[0016] A deformation operation is performed on the multi-scale features according to the key frame image to obtain the prior depth map.

[0017] Optionally, constructing a first Gaussian model according to the first camera pose, the image information, the sparse point cloud map, the prior depth map, and the current frame image and optimizing the first camera pose to obtain a second camera pose specifically includes:

[0018] Performing a first optimization on the prior depth map according to the sparse point cloud map, back-projecting the first optimized prior depth map and the image information using the first camera pose to generate a first Gaussian point cloud, and constructing a first Gaussian model based on the first Gaussian point cloud;

[0019] Rendering the first Gaussian model to obtain a rendered image, determining a first loss between the rendered image and the current frame image, and optimizing the first Gaussian model and the first camera pose using back propagation based on the first loss to obtain the second camera pose.

[0020] Optionally, obtaining a second Gaussian model according to the sparse point cloud map, the prior depth map, the image information and the second camera pose specifically includes:

[0021] Performing a second optimization on the prior depth map according to the sparse point cloud map, and back-projecting the second optimized prior depth map and the image information using the second camera pose to generate a second Gaussian point cloud;

[0022] A second Gaussian model is constructed according to the second Gaussian point cloud.

[0023] Optionally, the key frame image is updated by the following method:

[0024] Divide the current frame image into M regions, and determine matching feature points between each region and the key frame image of the current frame image and the number of the matching feature points; M is a natural number greater than 1;

[0025] The distribution uniformity is determined according to the number of the matching feature points in each area. If both the number of the matching feature points and the distribution uniformity meet preset conditions, the key frame image of the current frame image is updated using the current frame image to obtain the key frame image of the next frame image.

[0026] Optionally, the method further includes:

[0027] Determine a second loss between the Gaussian map of the current frame image and the Gaussian map of the previous frame image;

[0028] The second camera pose and the second Gaussian model of all frame images are jointly optimized according to the second loss to obtain a final Gaussian map.

[0029] On the other hand, the present invention provides a monocular vision-based map construction system, comprising a camera, a front-end module, a back-end module and an optimization module, wherein:

[0030] The camera is used to obtain the current frame image;

[0031] The front-end module is used to extract image information of the current frame image and determine a first camera posture;

[0032] The backend module is configured to obtain a key frame image and a sparse point cloud map, determine a priori depth map of the current frame image on the key frame image; construct a first Gaussian model based on the first camera pose, the image information, the sparse point cloud map, the priori depth map, and the current frame image, and optimize the first camera pose to obtain a second camera pose;

[0033] The optimization module is used to obtain a second Gaussian model based on the sparse point cloud map, the prior depth map, the image information and the second camera posture, and add the second Gaussian model to the Gaussian map according to the second camera posture.

[0034] On the other hand, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the aforementioned method when executing the computer program.

[0035] The implementation of the present invention includes the following beneficial effects: combining the Gaussian model and the prior depth map to optimize the camera pose and improve the accuracy of camera pose estimation; and using the Gaussian distribution to represent the scene in the image can more accurately represent the geometry and appearance information of the scene, so that the generated map has higher visualization quality and geometric accuracy; on the premise of obtaining the optimized second camera pose, the Gaussian model is reconstructed using the same sparse point cloud map, prior depth map and image information as the optimized camera pose, so as to achieve scale unification of camera pose estimation and Gaussian model establishment, enhance the tracking stability and robustness of the system in complex scenes, and jointly optimize the camera pose and map geometry through the differentiable rendering characteristics of the Gaussian model, reduce the number of iterations, thereby improving real-time performance without reducing accuracy, thereby improving the accuracy of map construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flowchart of the steps of a monocular vision-based map construction method provided by the present invention;

[0037] Figure 2 This is an end-to-end execution flow diagram of a monocular vision-based map construction method provided by the present invention;

[0038] Figure 3 This is a structural diagram of a monocular vision-based map construction system provided by the present invention;

[0039] Figure 4 It is a structural schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0041] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The singular forms "a," "the," and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0042] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0043] In some embodiments, as Figure 1 and Figure 2 As shown, Figure 1 It is a flowchart of the steps of the map construction method based on monocular vision. Figure 2 The present invention provides an end-to-end execution flow diagram of a monocular vision-based map construction method, which includes the following steps:

[0044] S100: Obtain a current frame image, a key frame image, and a sparse point cloud map.

[0045] Among them, the current frame image represents the image being processed; the key frame image represents a number of frame images with higher feature point matching, and a number means more than 1.

[0046] S200: Extract image information of the current frame image and determine a first camera posture.

[0047] The image information includes but is not limited to depth information or RGB information, and the depth information includes but is not limited to image geometric information such as the spatial position of objects in the scene (corresponding to the position μ of the Gaussian model) and uncertainty (corresponding to the covariance ∑).

[0048] Specifically, step S200 may include the following steps:

[0049] S210 , dividing the current frame image into several sub-images based on a deep learning algorithm, extracting initial information, and determining a third camera posture based on the initial information.

[0050] The deep learning algorithm may be, but is not limited to, a deep learning visual odometry method, such as DeepPatch Visual Odometry (DPVO). The initial information includes, but is not limited to, unoptimized depth information, coordinates of matching feature points, or RGB information. The plurality of sub-images represent at least two blocks.

[0051] Specifically, the coordinates of the matching feature points can be, but are not limited to, a large number of feature points extracted from the image through a feature extraction algorithm, such as the ORiented Brief (ORB) feature extraction algorithm and the Scale-Invariant Feature Transform (SIFT) algorithm. The feature points of the current frame image are matched with the feature points of the key frame image through a feature matching algorithm, such as the BF-matcher algorithm and the Fast Library for Approximate Nearest Neighbors (FLANN) algorithm, to obtain the matching feature points and their coordinates.

[0052] When determining the third camera pose, the coordinates of the feature points in the initial information are used, including the following cases:

[0053] S211. If the current frame image is a frame image before the Nth frame, obtain initial information of several frame images next to the current frame, and calculate a basic matrix based on the initial information of the current frame image and the image information of the several frame images next to the current frame; calculate an essential matrix based on the basic matrix and the intrinsic parameter matrix of the camera, decompose the essential matrix to obtain the third camera posture; wherein N is a natural number greater than or equal to 1.

[0054] Specifically, for example, if N is equal to 1, then if the current frame image is the first frame image, the coordinates of the feature points of the next several frame images of the current frame are obtained, such as the coordinates of the feature points of the second frame image and the third frame image.

[0055] Based on the coordinates of these feature points, the matching relationship between different frame images is determined, and the initial camera pose (the third camera pose) is calculated using the epipolar geometry principle. Let the camera pose transformation matrix from the i-th frame image to the i+1-th frame image be T i,i+1 , through epipolar geometry, using the coordinates p of the feature point pairs matched between the i-th frame image and the i+1-th frame image i and p i+1 (coordinates on the normalized plane) calculate the basic matrix F, as shown in formula (1):

[0056]

[0057] Among them, F is a 3×3 matrix that contains the essential geometric relationship between the two frames of images.

[0058] Then calculate the essential matrix E=K based on the basic matrix F and the camera's intrinsic parameter matrix K TFK, finally, by performing singular value decomposition (SVD) on the essential matrix E, it is decomposed into E = Udiag (1, 1, 0) V T , we can get the rotation matrix R and translation vector t, and then determine the third camera posture T of the first frame image i,i+1 =[R|t].

[0059] S212: If the current frame image is an image frame after the Nth frame, obtain the motion speed and time interval between the current frame image and the previous frame image, and determine a third camera posture according to the motion speed, time interval and camera posture of the previous frame image.

[0060] Among them, the previous frame image can be the frame among the key frame images that is closest to the current frame image, or it can be the previous frame image of the current frame; the camera posture of the previous frame image can be the third camera posture corresponding to the previous frame image or the second camera posture obtained after optimization.

[0061] Specifically, if N is equal to 1, the second frame and subsequent images are suitable for step S212. Combine the constant velocity motion model to predict the third camera pose of the current frame image. Assume that the camera pose of the previous frame image is T i-1,i , the motion speed is v, the time interval is Δt, then the third camera posture T of the current frame image i,i+1 The prediction formula is as follows (2):

[0062] T i,i+1 =T i-1，i ·Exp(vΔt) (2)

[0063] Among them, Exp(·) is the Lie group exponential mapping, which converts the velocity vector v into the corresponding transformation matrix. Then, the update operator is run to fine-tune and optimize the predicted camera pose to make it more consistent with the actual scene.

[0064] S220: Calculate the inter-block optical flow and reprojection error of the sub-images, process the inter-block optical flow and reprojection error using a nonlinear optimization method, update the third camera pose and the initial information, and obtain image information and the first camera pose. S300: Determine a priori depth map for the current frame image on the key frame image; construct a first Gaussian model based on the first camera pose, image information, the sparse point cloud map, the priori depth map, and the current frame image, and optimize the first camera pose to obtain a second camera pose.

[0065] The Gaussian model includes Gaussian distributions of several scenes, and each Gaussian distribution may include the position, rotation, scale, opacity or color of an object in the scene.

[0066] Specifically, a pre-trained Multi-View Stereo (MVS) network can be used on the keyframe image window to combine the current frame image and image information to estimate the prior depth map. The MVS network uses a Feature Pyramid Network (FPN) to extract multi-scale features of the current frame image and constructs a 2D cost volume based on the keyframe image and depth information through differentiable deformation operations to generate the prior depth map.

[0067] In step S300, constructing a first Gaussian model based on the first camera pose, image information, sparse point cloud map, prior depth map, and current frame image and optimizing the first camera pose to obtain a second camera pose may include the following steps:

[0068] S310: Perform a first optimization on the prior depth map according to the sparse point cloud map, back-project the first optimized prior depth map and image information using a first camera pose to generate a first Gaussian point cloud, and construct a first Gaussian model based on the first Gaussian point cloud.

[0069] Specifically, the sparse point cloud map is used to perform geometric alignment and scale correction on the prior depth map, remove noise and optimize depth information estimation, and obtain a first optimized prior depth map.

[0070] The first optimized prior depth map and RGB information are back-projected using the first camera pose to generate a first Gaussian point cloud. After downsampling the first Gaussian point cloud, a first Gaussian model is constructed.

[0071] S320: Render the first Gaussian model to obtain a rendered image, determine a first loss between the rendered image and the current frame image, and optimize the first Gaussian model and the first camera pose using back propagation based on the first loss to obtain a second camera pose.

[0072] The first loss includes, but is not limited to, photometric loss, depth geometry loss, depth smoothing regularization loss, and isotropic loss. The first Gaussian model and the second Gaussian model are both 3D Gaussian models. The Gaussian model includes several Gaussian distributions, each of which represents a scene. The parameters of the Gaussian distribution represent the position, rotation, scale, opacity, or color of objects in the scene.

[0073] Specifically, the first Gaussian model is projected onto the image plane through differentiable rendering to obtain the rendered image corresponding to the first Gaussian model. The photometric loss, depth geometry loss, depth smoothing regularization loss and isotropy loss of the rendered image and the current frame image (original image) are calculated. Based on these losses, the parameters of the first Gaussian model and the first camera pose are jointly optimized using back propagation to obtain the second camera pose.

[0074] S400: Obtain a second Gaussian model according to the sparse point cloud map, the prior depth map, the image information, and the second camera pose, and add the second Gaussian model to the Gaussian map according to the second camera pose.

[0075] Specifically, step S400 includes the following steps:

[0076] S410: Perform a second optimization on the prior depth map based on the sparse point cloud map, wherein the second optimization includes geometric alignment and scale correction. Back-project the second optimized prior depth map and RGB information in the image information using a second camera pose to generate a second Gaussian point cloud.

[0077] S420: Construct a second Gaussian model according to the second Gaussian point cloud.

[0078] Finally, the second Gaussian model is added to the Gaussian map according to the second camera pose. Steps S410-420 achieve scale alignment between the front-end tracking and the back-end Gaussian map, ensuring scale consistency and real-time performance of the entire system in different scenarios.

[0079] In step S400, the present invention further includes using the Gaussian map of the current frame image to render the depth map of the front-end tracking frame in real time and initialize the sparse point cloud map to achieve scale alignment.

[0080] Step S400 may be referred to as a sparse-dense adjustment ring (SDAR) strategy.

[0081] After step S400, the present invention may further include step S500 of determining a key frame image of the next frame image. Specifically, the step includes: dividing the current frame image into M regions, determining the matching feature points between each region and the key frame image of the current frame image and the number of matching feature points; M is a natural number greater than 1. The number of matching feature points includes the total number N 4,j and the number of each region n k .

[0082] According to the total number N 4,j and the number of each region n k Determine the distribution uniformity H, as shown in formula (3):

[0083]

[0084] In information theory, when calculating distribution uniformity (entropy), the base of the log(.) function is usually 2 or the natural logarithm e. The base does not affect the relative judgment of the uniformity of the feature point distribution, but only affects the numerical scale. The base can be determined based on actual calculation requirements.

[0085] If the number and distribution uniformity of matching feature points meet the preset conditions, for example, if the number of matching points N 4，j Reach a certain threshold N thresh , and the distribution uniformity H is greater than the uniformity threshold H thresh , add the current frame image to the key frame image of the current frame image to obtain the key frame image of the next frame image.

[0086] If the current frame image is the first frame image, the second frame image or the third frame image, the key frame image may be the latest three frame images as the key frame images.

[0087] If the current frame image is the fourth frame image or an image after the fourth frame image, the key frame image may be the key frame image updated in step S500 .

[0088] In this embodiment, after step S400, the method further includes constructing a global optimization model using the S600GTSAM library, specifically including:

[0089] Determine the second loss between the Gaussian map of the current frame and the Gaussian map of the previous frame. This second loss includes, but is not limited to, reprojection error and geometric depth smoothing loss. To minimize these two losses, the second camera poses and the second Gaussian model for all frames are jointly optimized to produce the final Gaussian map. This reflects scene changes and ensures positioning accuracy and map quality.

[0090] The reprojection error is calculated as follows:

[0091] The Gaussian map of the current frame image is matched with the Gaussian map of the previous frame image. Based on the matching results, the three-dimensional coordinates of the feature points in the scene are estimated using the epipolar geometry and triangulation principles. Let the normalized coordinates of the matched feature point pair in the i-th frame and the i+1-th frame be p i and p i+1 , calculate the unnormalized coordinates of the feature point P by triangulation, as shown in formula (4):

[0092] P1=K -1 p i λ i =K -1 p i+1 λ i+1 (4)

[0093] Among them, λ i and λ i+1 is the depth scale factor.

[0094] The reprojection error is calculated according to formula (5):

[0095] e rep =∑j ||K[R j |t j ]P1 j -p j || 2 (5)

[0096] Among them, R j and t j are the rotation and translation of the second camera pose of the j-th frame image, P1 j is the coordinate of the 3D point in the Gaussian map, p j is the projection point of the three-dimensional point on the image, and K is the intrinsic parameter matrix of the camera.

[0097] Step S500 may be performed after steps S100 to S400 are performed for all frames of images, or after steps S100 to S400 are performed for several frames of images. Figure 3 As shown, Figure 3 The present invention provides a structural diagram of a monocular vision-based map construction system, which includes a camera, a front-end module, a back-end module and an optimization module, wherein:

[0098] Camera, used to obtain the current frame image;

[0099] A front-end module, configured to extract image information of a current frame image and determine a first camera posture;

[0100] A backend module is configured to obtain a keyframe image and a sparse point cloud map, determine a priori depth map of the current frame image based on the keyframe image; construct a first Gaussian model based on the first camera pose, image information, the sparse point cloud map, the priori depth map, and the current frame image, and optimize the first camera pose to obtain a second camera pose;

[0101] The optimization module is used to obtain a second Gaussian model according to the sparse point cloud map, the prior depth map, the image information and the second camera posture, and add the second Gaussian model to the Gaussian map according to the second camera posture.

[0102] Specifically, the relevant term explanations and specific steps are the same as those in the aforementioned method embodiment.

[0103] The present invention can be used in the fields of visual tracking, navigation and path planning.

[0104] This method calculates the number and distribution of feature point matches with existing keyframes, determines whether the Gaussian common visibility criterion is met, and then determines whether the current frame image should be used as a new keyframe image. This strategy provides more stable and rich scene information for back-end mapping and optimization, helping to improve the accuracy of map construction and pose estimation.

[0105] In some embodiments, such as Figure 4 As shown, Figure 4 1 is a structural diagram of an electronic device provided by the present invention. The present invention also provides an electronic device, which includes a processor 10 and a memory 11, wherein the memory 11 stores a computer program, and when the processor 10 executes the computer program, it implements any one of the methods described in the above method embodiments.

[0106] Among them, the memory is a non-transient computer-readable storage medium that can be used to store non-transient software programs and non-transient computer executable programs. The memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory optionally includes a remote memory remotely arranged relative to the processor, and these remote memories can be connected to the processor via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0107] In order to better illustrate the beneficial effects of the present invention, the present invention provides the following experimental data:

[0108] Experimental setup:

[0109] TUM Dataset: The TUM dataset covers a variety of indoor scenes, such as offices and conference rooms, with varying lighting intensities, object placement, and camera motion patterns. For example, office scenes feature complex office equipment and diverse light reflections, while conference room scenes include large conference tables, projectors, and other facilities, enabling comprehensive testing of system performance in typical indoor environments.

[0110] Replica Dataset: This dataset contains high-precision 3D reconstruction models as real-world references, obtained through precise laser scanning and modeling. This dataset provides a reliable basis for accurately evaluating the system's reconstruction quality. It includes a variety of architectural structures and interior decorations, including diverse room layouts and complex texture details, helping to test the system's reconstruction accuracy for complex scenes.

[0111] ICL-NUIM dataset: Contains a variety of complex scenes and diverse camera motion trajectories, such as rapid movement, rotation, and movement under occlusion. It is used to rigorously test the adaptability and stability of the system in complex dynamic scenes.

[0112] Comparison method:

[0113] ORB-SLAM3: As a classic monocular vision SLAM algorithm, it uses the ORB (OrientedFAST and RotatedBRIEF) feature in the feature extraction stage, which is fast and stable. During tracking, it uses the bag-of-words model for rapid positioning and relocalization. However, when dealing with complex scenes and dynamic objects, the accuracy of its pose estimation and map construction is affected to a certain extent.

[0114] DSO-SLAM: A recently proposed deep learning-based monocular SLAM method estimates camera pose and scene depth by directly optimizing the photometric error of the image. However, this method is sensitive to changes in illumination, and its robustness under varying lighting conditions needs to be improved.

[0115] Evaluation Metrics:

[0116] Absolute trajectory error (ATE): used to evaluate the attitude estimation accuracy of the system, by calculating the estimated trajectory T est With the real trajectory T gt It is measured by the Euclidean distance between them, as shown in formula (6):

[0117]

[0118] Where N is the number of frames in the trajectory. A lower ATE value indicates that the estimated trajectory is closer to the true trajectory and the pose estimation accuracy is higher.

[0119] Peak signal-to-noise ratio (PSNR): used to evaluate the rendering quality of the new view, reflecting the difference in brightness between the reconstructed image and the real image, as shown in formula (7):

[0120]

[0121] Among them, MAX I is the maximum pixel value of the image, and MSE is the mean square error, which is the mean of the sum of the squares of the differences between the corresponding pixels in the reconstructed image and the true image. A higher PSNR value indicates that the brightness of the reconstructed image is closer to the true image.

[0122] Structural Similarity Index (SSIM): Also used to evaluate the quality of new view rendering, it focuses on measuring the structural similarity between the reconstructed image and the real image. The calculation formula is relatively complex, involving the brightness comparison function l(x,y), the contrast comparison function c(x,y), and the structure comparison function s(x,y), as shown in formula (8):

[0123] SSIM(x,y)=[l(x,y)] α [c(x,y)] β [s(x,y)] γ (8)

[0124] Among them, α, β, and γ are parameters used to adjust the weights of the three comparison functions, usually taking α = β = γ = 1. The closer the SSIM value is to 1, the more similar the structure of the reconstructed image is to the real image.

[0125] Learning Perceptual Patch Similarity (LPIPS): This metric evaluates the quality of geometric reconstruction and measures the similarity between the reconstruction and the real scene from a human perspective. This metric extracts image features using a neural network and calculates the distance between these features to assess similarity. Lower values indicate a greater perceptual similarity between the reconstruction and the real scene.

[0126] Experimental results show: quantitative data:

[0127] On the TUM dataset, the average ATE of the present invention is reduced by 20% compared to ORB-SLAM3 and by 15% compared to DSO-SLAM. For example, in a specific sequence, the ATE of ORB-SLAM3 is 0.25 meters, the ATE of DSO-SLAM is 0.22 meters, and the ATE of this system is only 0.2 meters, indicating that the accuracy of posture estimation has been significantly improved. In terms of new view rendering quality, the PSNR value is improved by 10% compared to ORB-SLAM3, and the SSIM value is improved by 8%. For example, in the reconstruction of an indoor scene image, the PSNR of ORB-SLAM3 is 25dB and the SSIM is 0.8, while the PSNR of this system reaches 27.5dB and the SSIM reaches 0.864, indicating that the reconstructed image is closer to the real scene.

[0128] Similar superior results were achieved on the Replica and ICL-NUIM datasets. In the complex architectural scenes of the Replica dataset, the system significantly outperformed the competing algorithms in terms of map completeness and accuracy. In the dynamic scenes of the ICL-NUIM dataset, the system maintained high pose estimation accuracy and map reconstruction quality, demonstrating its stability and reliability in complex scenarios.

[0129] Result analysis:

[0130] The role of joint optimization: The joint optimization of Gaussian mapping and sparse point cloud allows pose estimation and map construction to be mutually constrained. During the pose estimation process, the scene information provided by the first Gaussian model can help correct pose deviations; accurate pose estimation can provide a more reliable basis for updating the Gaussian map, thereby improving overall accuracy. For example, in complex indoor scenes, when encountering occlusion or missing feature points, the joint optimization mechanism can leverage the existing Gaussian map information and local tracking information from the image information to maintain the continuity and accuracy of pose estimation.

[0131] Contribution of Depth Estimation: Depth estimation based on the MVS network provides accurate geometric constraints for the Gaussian map. By fusing features from multi-view images, the MVS network can more accurately estimate scene depth, allowing the Gaussian model in the Gaussian map to more accurately represent the position and shape of scene objects, improving the geometric reconstruction quality of the map. In scenes with complex geometric structures, such as stairs and furniture, depth estimation based on the MVS network can generate a more realistic Gaussian map, greatly improving map accuracy compared to traditional methods.

[0132] The effectiveness of the loss method: The geometric depth smoothing loss method effectively reduces the impact of depth estimation errors on the map. By smoothing the depth map, noise and outliers are removed, making the construction of the Gaussian map more stable and accurate. In real-world scenarios, when depth estimation fluctuates due to factors such as lighting changes or surface reflections, the geometric depth smoothing loss method can optimize the depth map to ensure that the quality of the Gaussian map is not affected.

[0133] Advantages of the SDAR strategy: The SDAR strategy achieves efficient collaboration between the front-end and back-end. The sparse point cloud map extracted by the front-end corrects the prior depth map estimated by the MVS network, removing noise and errors in the depth estimation; the optimized prior depth map generates a new Gaussian model and adds it to the Gaussian map, further enriching the map information; the real-time rendering capability of the Gaussian map is used to achieve scale alignment between the front-end tracking and the back-end Gaussian map, ensuring the scale consistency and real-time performance of the entire system in different scenarios. In dynamic scenarios, the SDAR strategy can quickly adjust the system to adapt to scene changes and maintain stable tracking and map building performance.

[0134] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A map construction method based on monocular vision, characterized in that: The following steps are involved: Get the current frame image, key frame image and sparse point cloud map; Extracting image information of the current frame image and determining a first camera posture; Determining a priori depth map of the current frame image on the key frame image; Constructing a first Gaussian model based on the first camera pose, the image information, the sparse point cloud map, the prior depth map, and the current frame image, and optimizing the first camera pose to obtain a second camera pose; wherein the Gaussian model includes Gaussian distributions of multiple scenes; A second Gaussian model is obtained according to the sparse point cloud map, the prior depth map, the image information, and the second camera pose, and the second Gaussian model is added to the Gaussian map according to the second camera pose.

2. The method according to claim 1, characterized in that The extracting image information of the current frame image and determining the first camera posture specifically includes: Divide the current frame image into several sub-images based on a deep learning algorithm, extract initial information, and determine a third camera posture based on the initial information; An inter-block optical flow and a reprojection error of the sub-image are calculated, and the inter-block optical flow and the reprojection error are processed using a nonlinear optimization method, and the third camera pose and the initial information are updated to obtain the image information and the first camera pose.

3. The method according to claim 2, characterized in that The determining of the third camera posture according to the image feature specifically includes: If the current frame image is an image frame before the Nth frame or the Nth frame image, obtaining initial information of several frames of images next to the current frame, and calculating a fundamental matrix based on the initial information of the current frame image and the image information of the several frames of images next to the current frame; calculating an essential matrix based on the fundamental matrix and an intrinsic parameter matrix of the camera, and decomposing the essential matrix to obtain the third camera pose; wherein N is a natural number greater than or equal to 1; If the current frame image is a frame image after the Nth frame, the motion speed and time interval between the current frame image and the previous frame image are obtained, and the third camera posture is determined according to the motion speed, the time interval and the camera posture of the previous frame image.

4. The method according to claim 1, wherein The determining of the prior depth map of the current frame image on the key frame image specifically includes: Extracting multi-scale features of the current frame image; A deformation operation is performed on the multi-scale features according to the key frame image to obtain the prior depth map.

5. The method according to claim 1, wherein The step of constructing a first Gaussian model according to the first camera pose, the image information, the sparse point cloud map, the prior depth map, and the current frame image and optimizing the first camera pose to obtain a second camera pose specifically includes: Performing a first optimization on the prior depth map according to the sparse point cloud map, back-projecting the first optimized prior depth map and the image information using the first camera pose to generate a first Gaussian point cloud, and constructing a first Gaussian model based on the first Gaussian point cloud; Rendering the first Gaussian model to obtain a rendered image, determining a first loss between the rendered image and the current frame image, and optimizing the first Gaussian model and the first camera pose using back propagation based on the first loss to obtain the second camera pose.

6. The method according to claim 1, characterized in that The obtaining of a second Gaussian model according to the sparse point cloud map, the prior depth map, the image information and the second camera pose specifically includes: Performing a second optimization on the prior depth map according to the sparse point cloud map, and back-projecting the second optimized prior depth map and the image information using the second camera pose to generate a second Gaussian point cloud; A second Gaussian model is constructed according to the second Gaussian point cloud.

7. The method according to claim 1, characterized in that The key frame image is updated by the following method: Divide the current frame image into M regions, and determine matching feature points between each region and the key frame image of the current frame image and the number of the matching feature points; M is a natural number greater than 1; The distribution uniformity is determined according to the number of the matching feature points in each area. If both the number of the matching feature points and the distribution uniformity meet preset conditions, the key frame image of the current frame image is updated using the current frame image to obtain the key frame image of the next frame image.

8. The method according to claim 1, characterized in that The method further comprises: Determine a second loss between the Gaussian map of the current frame image and the Gaussian map of the previous frame image; The second camera pose and the second Gaussian model of all frame images are jointly optimized according to the second loss to obtain a final Gaussian map.

9. A map construction system based on monocular vision, characterized in that: It includes camera, front-end module, back-end module and optimization module, among which, The camera is used to obtain the current frame image; The front-end module is used to extract image information of the current frame image and determine a first camera posture; The backend module is configured to obtain a key frame image and a sparse point cloud map, determine a priori depth map of the current frame image on the key frame image; construct a first Gaussian model based on the first camera pose, the image information, the sparse point cloud map, the priori depth map, and the current frame image, and optimize the first camera pose to obtain a second camera pose; The optimization module is used to obtain a second Gaussian model based on the sparse point cloud map, the prior depth map, the image information and the second camera posture, and add the second Gaussian model to the Gaussian map according to the second camera posture.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

Images