A target building visual localization method based on generative dense model

Abstract

The present invention proposes a method for visual positioning of a target building based on a generative dense model. In response to the problem that traditional visual positioning methods are heavily dependent on expensive three-dimensional reconstructed maps, the present invention designs a two-stage positioning method. In the offline registration stage, the 3D dense model obtained by the generative model is used as input, and a joint pose-scale optimization algorithm is used to optimize the geometric proportions of the 3D model and register the rendered images and their corresponding poses under multiple perspectives; in the online positioning stage, the present invention can quickly obtain the camera pose corresponding to the query image based on a pre-trained lightweight cross-domain feature matcher and a robust pose solver. The present invention takes into account both positioning accuracy and speed, and can be deployed and used in strict environments without maps and few references, which can significantly improve the effect of low-quality models in visual positioning. Compared with existing visual positioning methods, the present invention shows wider applicability and more accurate positioning results.

Description

Technical Field

[0001] The present invention relates to the fields of augmented reality, visual positioning and generative visual models, and in particular to a target building visual positioning method based on a generative dense model. Background Art

[0002] Augmented Reality (AR) is a new technology that integrates real-world information with virtual image information. This technology uses computer technology to simulate real-world information, then overlays virtual information onto the real world, creating a new experience and application that transcends reality. With the development of computer vision technology, particularly structure-from-motion mapping (SfM), simultaneous localization and mapping (SLAM), and visual localization (Visual Localization), many AR applications targeting outdoor scenes have emerged. Visual localization involves calculating the camera position and posture (i.e., the camera pose) of the captured image based on relevant image information captured by a camera or other visual sensor. The camera pose derived from visual localization can be used to generate corresponding AR effects.

[0003] Classic visual localization solutions for outdoor scenes typically require extensive image acquisition and calibration of the real scene. A 3D point cloud map of the scene is then constructed using offline or simultaneous mapping techniques. During the localization phase, visual localization algorithms or sensor fusion are used to infer the camera pose of the user capturing the scene at a specific location. However, due to the significant overhead and storage consumption of scene mapping, this approach is typically costly to deploy. For larger outdoor scenes or buildings, obtaining a point cloud map that is reliable for localization typically requires extensive, high-quality data acquisition and calibration. The mapping time increases exponentially with the scale of the reconstruction required. To address the cost of mapping, visual localization methods based on dense 3D models (such as meshes) have been proposed, such as the deep learning-accelerated visual localization technique based on mesh representation proposed by Zhang P in "DLALoc: Deep-Learning Accelerated Visual Localization Based on Mesh Representation." This method eliminates the need for complex image acquisition and point cloud reconstruction, instead basing localization on readily available or low-precision mesh models, significantly reducing mapping time and computational costs.

[0004] There are still some shortcomings in the existing visual positioning solutions based on 3D dense models. First, for the acquisition of 3D models, CAD models are generally downloaded from the Internet, or downloaded from open source vector map tools (such as Google Earth). This is an unstable resource acquisition channel. Once the required resources are not available, manual production or re-mapping is still required. Secondly, since most 3D dense models cannot be correctly aligned or scaled with the real scene, there are usually large errors in the geometric authenticity of the model, which will lead to the difficulty of converging to the correct result due to the large 3D point error when positioning the query image. Third, since the appearance of the 3D dense model is different from the real scene, that is, there is a large error in the appearance authenticity of the model, traditional local feature solutions have difficulty matching models with huge differences in feature distribution with query images, or require the use of a large and expensive dense matching algorithm. The above problems are severe challenges to the availability, accuracy and efficiency of existing positioning algorithms.

[0005] In summary, traditional solutions have problems such as high mapping overhead, difficulty in obtaining available dense models, and low model quality leading to low positioning accuracy and efficiency. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention proposes a target building visual positioning method based on a generative dense model.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for visually localizing a target building based on a generative dense model includes the following steps:

[0009] Step 1: Collect an undistorted single-view image of the target building. Use the sparse multi-view generative diffusion model to expand the single-view image to generate images of the target building and its poses from multiple perspectives. The expanded image serves as the reference image for subsequent steps.

[0010] Step 2: Based on all reference images, a 3D dense model is generated using the reconstructed large model based on sparse views;

[0011] Step 3: Based on each reference image and its pose, render a rendered depth map with the same pose in the 3D dense model; perform monocular depth estimation on the reference image to obtain an estimated depth map; based on the rendered depth map and the estimated depth map, jointly optimize the scale of the 3D dense model and the pose of each reference image to obtain the optimized 3D dense model and the pose of each reference image;

[0012] Step 4: Obtain a query image of the target building to be located, select the reference image that is most similar to the query image from all reference images, retrieve the optimized pose corresponding to the reference image, and render the selected reference image in the optimized 3D dense model based on the retrieved pose to obtain a reference depth map;

[0013] Step 5: Feature extraction and 2D-2D cross-domain matching are performed on the query image and the filtered reference image to obtain matching feature point pairs. The feature points belonging to the reference image in the matching feature point pair are promoted to 3D using the reference depth map. The promoted feature points and the feature points belonging to the query image in the matching feature point pair form 2D-3D feature point pairs. The 2D-3D feature point pairs are used to solve the camera pose through the 6D pose solver to obtain the pose of the query image.

[0014] Furthermore, in step one, the sparse multi-perspective diffusion model is used to expand the single-perspective image to generate images of the target building and its posture under multiple perspectives. Specifically, the single-perspective image is input into the Zero-1-to-3 model, 6 evenly distributed initial postures are set, and the initial images corresponding to the initial postures are generated by the Zero-1-to-3 model. The initial images are then optimized and constrained by the ConsistNet module so that the 6 initial images meet the geometric consistency assumption. The optimized and constrained images and their corresponding initial postures are the images and postures of the target building under multiple perspectives.

[0015] Furthermore, in step 2, the large reconstructed model based on sparse views is a FS3D pre-trained model.

[0016] Furthermore, in step three, the rendering in the 3D dense model to obtain a rendered image and its depth map with the same posture is performed using a rasterization rendering method or a projection algorithm.

[0017] Furthermore, in step 3, the scale ratio of the 3D dense model and the pose of each reference image are jointly optimized based on the rendered depth map and the estimated depth map, specifically including:

[0018] 1) According to the rendered depth map and the estimated depth map, the 2D pixel points of the area corresponding to the target building in the reference image are extracted, and the 2D pixel points are upscaled according to the rendered depth map and the estimated depth map to obtain two sets of 3D point clouds. in and is the 3D point obtained from the rendered depth map and the estimated depth map, Represents a three-dimensional point set;

[0019] 2) The principal component analysis algorithm is used to calculate the eigenvalues ​​and eigenvectors corresponding to the first three principal components of the two groups of 3D point clouds respectively. The root mean square of the eigenvalues ​​obtained from the same group of 3D point clouds is solved as the scale coefficient of the 3D point cloud. The ratio of the scale coefficients of the two groups of 3D point clouds is calculated to obtain the prior scaling factor. The prior scaling factor is applied to the 3D point corresponding to the rendered depth map. On top, a set of pre-adjusted 3D point clouds is obtained;

[0020] 3) Use the pose of the current reference image as the initial pose p;

[0021] 4) Construct the three-dimensional scaling matrix S of the point cloud scale in the world coordinate system w =diag(s x ,s y ,s z ), where s x ,s y ,s z The x, y, and z scaling ratios of the point cloud are used to construct an optimization function E(p,S w ), iteratively optimize the pose p and three-dimensional scaling matrix S according to the optimization function w , until the convergence condition is reached, and the optimized pose p* and the optimized homogeneous two-dimensional scaling matrix S are obtained. w * ; The optimization function E(p,S w )for:

[0022]

[0023] Among them, W ij is the weight, β is a hyperparameter, λ is a regularization weight, I is the identity matrix, and ‖·‖ is the L1 norm;

[0024] 5) Repeat steps 1)-4) for each reference image to obtain multiple optimized poses p * and the 3D scaling matrix The geometric mean of the optimized three-dimensional scaling matrices is calculated element by element to obtain the final three-dimensional scaling matrix. According to the final three-dimensional scaling matrix The 3D dense model is scaled by the scaling ratios in the x, y, and z directions to obtain the optimized 3D dense model M * .

[0025] Furthermore, in step 5, the feature extraction and 2D-2D cross-domain matching of the query image and the selected reference image are performed to obtain matching feature point pairs, specifically including:

[0026] a) Using the visual basis model to analyze the query image IA and the filtered reference image I B Perform image encoding to obtain image visual feature maps

[0027] b) Using lightweight feature matching to query image I A and reference image I B The feature points p of the query image and the reference image are extracted respectively A ,p B , and calculate the descriptor d of the feature point A ,d B ;

[0028] c) For query image I A and reference image I B , calculate the feature similarity of every two feature points in the image in the corresponding image visual feature map;

[0029]

[0030] Among them, p i and p j For any feature point in the query image or reference image, F c (·) represents the visual feature vector of the feature point in the corresponding image visual feature map, Sim(p i ,p j ) is the feature point p i and p j The feature similarity between

[0031] d) Take the query image I A The feature points are nodes to establish an undirected feature point graph in the image Take reference image I B The feature points are nodes to establish an undirected feature point graph in the image Establish a directed feature point graph between images with the feature points of the query image and the feature points of the reference image as nodes and

[0032] e) Based on the feature similarity between feature points, the self-attention mechanism and the cross-attention mechanism are used to propagate the edges in the undirected feature point graph within the image and the directed feature point graph between images, and the descriptors of the nodes are updated during the propagation process; based on the descriptors of the updated nodes, the query image I A The feature points and reference image I B Establish an association matrix between the feature points:

[0033]

[0034] in is the descriptor of the i-th feature point in the query image, is the descriptor of the jth feature point of the reference image, C i,j is the degree of association;

[0035] f) The correlation degree is greater than the preset threshold ∈ c The query image I A The feature points and reference image I B The feature point combination is taken as the matching feature point pair.

[0036] Furthermore, in step five, the 6D pose solver includes two parts: basic pose solution and bundle adjustment optimization; the 2D-3D feature point pairs are used to solve the camera pose through the 6D pose solver as follows: the 2D-3D feature point pairs are input into the basic pose solution part, which obtains the matching confidence of the 2D-3D feature point pairs, and solves the basic pose of the 2D-3D feature points using the RANSAC-based P3P algorithm under the guidance of the matching confidence; the obtained basic pose is optimized using the bundle adjustment method, and the optimized pose is the pose of the query image.

[0037] The beneficial effects of the present invention are:

[0038] 1) Regarding the acquisition of dense 3D models, this paper leverages advanced 3D asset generation technology, based on the theoretical foundation of reconstructing large models from sparse perspectives, and combines robust multi-perspective image generation to rapidly obtain a basic dense 3D model as a positioning asset. The generated dense 3D model addresses the mapping dependency and missing 3D model sources issues of visual positioning, making it easy to deploy.

[0039] 2) Regarding the quality of 3D dense models, this invention optimizes them from two perspectives. First, during generation, multi-view image supervision is used to ensure the degree of model restoration as much as possible. Second, this invention considers using sparse view reference images combined with model parameters for joint optimization. The joint pose-scale optimization method uses an iterative optimization strategy to simultaneously optimize the ratio of 2D images to 3D models, solving the problem of poor collective authenticity of 3D dense models and effectively obtaining usable map data for visual positioning.

[0040] 3) Regarding the robustness of online positioning, the present invention utilizes an efficient local feature matching algorithm guided by a visual base model and can be applied to cross-domain image matching to quickly obtain a robust correspondence between the real query image and the generated reference image, thereby performing high-precision pose estimation. The cross-domain local feature matching algorithm can perform cross-domain matching, so there is no need to render the 3D dense model during the online positioning stage, and there is no need to obtain real pre-annotated building images as positioning map database images, thereby improving the efficiency and robustness of positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the overall framework diagram of the present invention;

[0042] Figure 2 Schematic diagram of the cross-domain matching network in the present invention;

[0043] Figure 3 It is a line graph of the experimental accuracy of the cross-domain matching network in the present invention;

[0044] Figure 4 This is a time-accuracy scatter plot comparing the present invention with other methods;

[0045] Figure 5 It is a rendering of the 3D dense model reconstruction of the target building of the present invention;

[0046] Figure 6 This is an effect diagram of the application of the present invention in AR technology. DETAILED DESCRIPTION

[0047] The present invention is a target building visual positioning technology based on a generative dense model. In the offline stage, a single or a small number of target building images are input to generate a 3D dense model, generate a reference image, and optimize the model. In the online stage, a query image and internal parameters are input to output the corresponding camera 6-DOF pose.

[0048] This method is implemented based on Python and C++. Figure 1 As shown, this method can be divided into an optimization phase and a positioning phase. The optimization phase requires deployment and processing on a device with GPU computing resources, while the positioning phase has lower requirements for GPU computing power and video memory. The optimization phase can be divided into three modules: the 2D generation module corresponds to step 1, the 3D generation module corresponds to step 2, and the optimized registration module corresponds to step 3. The online phase can be divided into three modules: the feature extraction module and the cross-domain matching module correspond to step 4, and the pose solution module provides the final output of the method.

[0049] The present invention will be described in detail below with reference to the accompanying drawings. The technical features of each embodiment of the present invention may be combined accordingly without conflict.

[0050] The main steps of the method of the present invention are as follows:

[0051] Step 1: First, collect images of the target building from a relatively complete observation angle. The collected single-view image without distortion is recorded as Then, the diffusion model F is generated using sparse multi-view I , generate a set of multi-view images of several target buildings And the corresponding posture Where N is the number of viewing angles, usually N = 6;

[0052] The sparse view generative diffusion model is a pre-trained model that accepts a single view image I0 as input and independently and individually models the conditional edge distribution of the image to generate a new set of images of 6 views with uniform view distribution. To ensure the geometric and appearance consistency of the generated image. The model uses linear noise scheduling to further improve the image consistency between multiple views. The model uses two-stage training and scaled reference attention mechanism to further improve the model's performance in the semantic content and texture details of the input image in the generated image. In order to adapt to the application scenario, the model is fine-tuned in the Warehouse public model dataset. Architectural models are collected from the Warehouse dataset and adjusted to the unit cube of [-0.5, 0.5]. The polar angle θ and azimuth angle are uniformly sampled from the spherical coordinate system according to the following formula And the radial distance r generates the rendering perspective, and the rendered image is used as the supervision truth value:

[0053]

[0054] In the specific implementation, the sparse multi-view diffusion network F IThe network is pre-trained using the Zero-1-to-3 model (Liu R, Wu R, Van Hoorick B, Tokmakov P, Zakharov S, Vondrick C. Zero-1-to-3: Zero-shot one image to 3D object. In Proceedings of the IEEE / CVF international conference on computer vision 2023 (pp. 9298-9309).). Image I0 is fed into the Zero-1-to-3 model, which generates six initial images from uniformly spaced viewpoints. After the initial images are generated, the six views must adhere to the geometric consistency assumption, meaning that each pixel's back-projection into the feature volume must be consistent. Based on this principle, the ConsistNet module (Yang J, Cheng Z, Duan Y, Ji P, Li H. Consistnet: Enforcing 3d consistency for multi-view images diffusion. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 7079-7088).) is connected after the Zero-1-to-3 model to optimize the constraints. The ConsistNet module first projects the view onto the global 3D voxel grid and infers the consistency. Then, it samples the light in the global 3D voxel to obtain consistent 3D features and aggregates them into each view to supervise the consistency.

[0055] Step 2: Obtain a multi-view image set Then, the large model F is reconstructed based on sparse views. M , generate a 3D dense model (3D mesh or dense point cloud) M;

[0056] The large reconstruction model based on sparse views is a pre-trained model. The model accepts a multi-view image set Based on the transformer backbone network, the image is mapped into an implicit triplane structure with multi-view supervision, and a dense 3D mesh model M is generated by FlexiCube or Splatting. The training of this model is based on the visual transformer encoder and the triplane encoder, and is supervised in two stages from implicit training to mesh training. In the specific implementation, the reconstructed large model F based on the sparse view is MThe FS3D (Zhu J, Ma H, Chen J, et al. Few-shot 3D shape generation[J]. arXiv preprint arXiv:2305.11664, 2023.) pre-trained model was used.

[0057] Step 3: Get an image collection Initial pose set After obtaining the dense 3D model M, a pose-scale joint optimization algorithm is used to simultaneously optimize the image pose and the scale ratio of the model, making the fit between the image and the model and the geometric accuracy of the model more accurate, and obtaining the optimized pose set and the 3D model after non-rigid scaling;

[0058] The pose-scale joint optimization algorithm described in step 3 is the core of the present invention, and the specific steps are as follows:

[0059] (3.1) For each input reference image I and corresponding pose p, the corresponding rendered depth map is rendered in the 3D grid M The rendering method here can use simple rasterization rendering or projection algorithm;

[0060] (3.2) For each input reference image I, use the monocular depth estimation model to obtain the corresponding estimated depth map

[0061] (3.3) Through the two depth maps, the target building parts in the image are lifted respectively to obtain two sets of 3D point clouds in and 3D points obtained for rendering depth and estimated depth, Represented as a three-dimensional point set;

[0062] (3.4) Perform principal component analysis (PCA) on the two groups of point clouds to decompose the eigenvalues ​​and eigenvectors corresponding to the first three principal components. The eigenvalues ​​are then root-mean-squared and used as the scale coefficient of the point cloud. The a priori scaling factor of the two groups of points is calculated by calculating the ratio and applied to the 3D points corresponding to the rendered depth. On, make it The scale difference is within an acceptable range; apply the a priori scaling factor to the 3D points The above refers to the prior scaling factor and the three-dimensional point Multiply to update the 3D point

[0063] (3.5) Construct an optimization function, use least squares optimization to optimize the function, and add the three-dimensional scaling matrix of the point cloud scale in the world coordinate system. The world coordinate system is the coordinate system where the dense model is located, and the three-dimensional scaling matrix is ​​the diagonal matrix S w =diag(s x ,s y ,s z ), where s x ,s y ,s z The x, y, and z scale ratios of the model in the world coordinate system. Based on the optimization function, the pose p and the scale matrix are iteratively optimized until convergence conditions are reached. The optimization objects are as follows:

[0064]

[0065] Among them, W ij is the weight, which is determined by the distance between the points. β is a hyperparameter used to control the influence of the squared distance term on the weight, λ is the regularization weight, I is the identity matrix, and ‖·‖ is the L1 norm;

[0066] (3.6) Repeat steps 1)-4) for each reference image to obtain multiple optimization results p * , And calculate the geometric mean of the optimized multiple three-dimensional scaling matrices element by element to obtain the final three-dimensional scaling matrix The final three-dimensional scaling matrix Applied to the 3D dense model M, the optimized model M is obtained * The scaling ratio S w The specific method applied to the 3D dense model M is to transform all vertex coordinates of the 3D dense model and the scaling factor S w Multiply.

[0067] In practice, step (3.1) can render the 3D mesh M using a rasterizer based on OpenGL for fast rendering results, or it can focus on quality and use a rendering engine such as Blender for more realistic rendering results. Since rendering only occurs during the optimization phase, the latter option can be chosen to improve the quality of feature points extracted in subsequent steps.

[0068] In practice, step (3.2) performs monocular depth estimation on an image using the monocular depth estimation model Depth Anything (Yang L, Kang B, Huang Z, et al. Depth anything: Unleashing the power of large-scale unlabeled data [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 10371-10381.). This model accepts a single image as input and directly outputs a relative depth image. Alternatively, a DPT model (dense prediction transformer) can be used.

[0069] The convergence condition of step (3.5) in the joint optimization algorithm is as follows: set a convergence threshold ∈, when E(p,S w )<∈, the model and image at that viewpoint are considered to have been well aligned and fitted, and the optimization is complete. Or when the number of iterations reaches 20, the algorithm is considered to have reached a local optimum and the optimization is stopped.

[0070] In the joint optimization algorithm, the regularization parameter λ is typically set to 1.0 for the optimization target in step (3.5). If the initial model differs significantly from the actual target building, the regularization parameter λ can be reduced to 0.1. To achieve automatic adaptation, a smaller regularization parameter is typically used in the first iteration, followed by a larger one.

[0071] In the joint optimization algorithm, the weight hyperparameter β is typically set to 0.1 for the optimization target in step (3.5). If the initial model differs significantly from the actual target building, the weight hyperparameter β can be reduced to 0.01. To achieve automatic adaptation, a smaller weight hyperparameter is typically used in the first iteration, followed by a larger weight hyperparameter.

[0072] Step 4: Obtain a query image of the target building to be located, select the reference image most similar to the query image from all reference images, retrieve the optimized pose and 3D dense model corresponding to the reference image, render the filtered reference image in the retrieved 3D dense model based on the retrieved pose, and obtain a reference depth map.

[0073] In step 4, selecting the reference image most similar to the query image is achieved through a global feature module, such as the MixVPR algorithm (A.Ali-Bey, B.Chaib-Draa and P.Giguére, "MixVPR: Feature Mixing for Visual Place Recognition," 2023IEEE / CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2023, pp. 2997-3006, doi:10.1109 / WACV56688.2023.00301.). Specifically, the global feature module extracts global features from each reference image and the query image, calculates the cosine similarity of the feature vectors between the global features of each reference image and the global features of the query image, and selects the reference image with the highest similarity.

[0074] Step 5: In the online positioning stage, a lightweight cross-domain feature matching algorithm is used to extract features and perform 2D-2D cross-domain matching on the query image and the most similar reference image, and then it is promoted to 3D through the reference depth map. The camera pose is then solved through a robust 6D pose solver (PnP algorithm and bundle adjustment optimization).

[0075] The lightweight cross-domain feature matching algorithm described in step 5 is implemented by a pre-trained model that accepts two images with significantly different statistical distributions (e.g., synthetic images and real images) I A ,I B , perform coarse matching through the DINOv2 base model, and perform fine matching through the cross attention mechanism, and finally output two sets of corresponding feature point coordinates The algorithm is established as a two-stage approach, such as Figure 2 As shown, the details are as follows:

[0076] 5.1) The query image I A and the reference image I with the highest similarity B Input, for the input image I A ,I B , using the DINOv2 model for image encoding to obtain robust coarse-precision image visual feature maps

[0077] 5.2) For image I A ,I B The feature points p of the image are extracted using the XFeat algorithm A ,p B , and calculate the descriptor d of the feature pointA ,d B , obtain a set of coarse-precision feature points and descriptors {d A ,p A},{d B ,p B};

[0078] 5.3) Utilizing visual features The feature similarity matrix is ​​established for every two feature points and descriptors, taking the query image to the reference image as an example:

[0079]

[0080] in, represents the visual feature vector at a certain position, is the i-th feature point of the query image, is the jth feature point of the query image, Feature points and The feature similarity between .

[0081] 5.4) To query the feature point p of the image A As nodes, connect any two nodes to establish an undirected feature point graph in the image Take the feature point p of the reference image B As nodes, connect any two nodes to establish an undirected feature point graph in the image Take the query image feature points and the reference image feature points as nodes, and connect the query image node p A and the node p of the reference image B , establish a directed feature point graph between images and picture The direction of the middle edge is from p A to p B ,picture The direction of the middle edge is from p B to p A ; The weight of the edge connecting two nodes in the feature point graph is the feature similarity between the two nodes.

[0082] 5.5) For both undirected and directed feature point graphs, a position-encoded self-attention mechanism propagates edges within the undirected feature point graph within an image, while a cross-attention mechanism propagates edges between directed feature point graphs between images. During this propagation process, the descriptor features of the nodes are updated. Specifically, each node in the graph receives the descriptor features of its neighboring nodes and calculates attention weights based on feature similarity, which are then used as new node features.

[0083] The above propagation process is repeated 3 times to obtain the optimized feature point graph and the final updated node descriptor features in the feature point graph.

[0084] 5.6) Directed feature point graph between optimized images Updated node descriptor features of feature points corresponding to the common part Establish the correlation matrix:

[0085]

[0086] Retain the correlation matrix with a correlation greater than the threshold ∈ c The point pairs are filtered and the corresponding feature point correspondence is used as the matching feature point pair, and the corresponding feature point correlation degree in the correlation matrix is ​​output as the matching confidence.

[0087] Furthermore, the feature points belonging to the reference image in the updated matching feature point pair are lifted to 3D by referring to the depth map, and the lifted feature points and the feature points belonging to the query image in the original matching feature point pair form a 2D-3D feature point pair.

[0088] Furthermore, the robust 6D pose solver described in step five includes two parts: RANSAC sampling based on matching confidence prior and PnP algorithm basic pose solution, and bundle adjustment optimization based on reprojection error. Specifically, the 2D-3D feature point pairs are input into the first part, RANSAC sampling is performed based on the matching confidence prior of the feature point pairs, and the sampled samples are solved by the PnP algorithm to obtain the basic pose; the solved basic pose is subjected to bundle adjustment optimization in the second part, and the optimized pose is the final pose of the query image. The matching confidence corresponds to the Gaussian correlation score of the feature point pair obtained previously. When solving, different matching pairs will be weighted according to the confidence, thereby guiding the RANSAC sampling.

[0089] In the specific implementation of the lightweight cross-domain feature matching algorithm described in step 5, the DINOv2 (ViT-S) model was selected to extract robust coarse-precision visual features of the image to balance time and accuracy.

[0090] In order to further demonstrate the implementation effect of the present invention, the present invention is compared with previous positioning solutions and processes in multiple aspects, two experiments are conducted, and actual AR effects are demonstrated.

[0091] The present invention is compared with the baseline solution on the CADLoc dataset. The CADLoc dataset contains landmark buildings of four classic tourist attractions (Notre Dame de Paris, the Pantheon, the Reichstag, and St. Peter's Square). Each building contains 3D mesh models of different qualities for positioning. The model quality includes high-quality multi-perspective geometric reconstruction results, as well as low-quality models manually reconstructed by artists, and even CAD models without textures. In order to test the positioning effect of the present invention on the generative model, a 3D mesh obtained based on the generative model was added to each scene in the experiment for testing. The experimental environment is Ubuntu 18.04, and the experiment was carried out under a GTX1080 graphics card (12G video memory). In the offline registration stage, the maximum video memory consumption is about 8G (dense matching RoMa peak requires 12G), and the video memory consumption in the online positioning stage is generally less, requiring only less than 4G, and can even be completely based on the CPU.

[0092] Experiment 1: Comparison of positioning accuracy and success rate

[0093] Figure 3 The positioning reprojection error recall rates of the positioning method of the present invention and the existing grid-based visual positioning method (MeshLoc) for four 3D models of different qualities (corresponding to A, B, C, and D) in the same landmark scene are demonstrated. The closer the curve is to the upper side, the higher the recall rate and the better the effect under the same error threshold. The present invention has significantly improved the absolute accuracy of positioning and the recall rate of dense reprojection error (Dense Re-Projection Error, DCRE), especially in the positioning of low-quality models (C, D). In the results of the generative model, the positioning results of this solution are also significantly better than the baseline solution, and are close to the positioning results reconstructed by MVS (multi-view geometry). This experiment proves the feasibility of this solution and the guarantee of AR effect.

[0094] Experiment 2: Comparison of online positioning efficiency

[0095] Figure 4 This study compared the speed and accuracy of our solution and the baseline solution during the online localization phase using different feature matching schemes. The horizontal axis represents the number of image pairs processed per second, and the vertical axis represents matching accuracy. By reducing dense feature dependencies and rendering and retrieval time, our solution significantly improves operational efficiency. This experiment demonstrates the effectiveness of our solution in practical applications.

[0096] The practical effects of the present invention are as follows Figure 5 and Figure 6 As shown: Through the dense model generation and joint optimization method of the present invention, a 3D dense model of the landmark building is reconstructed from a single-view image, as shown in Figure 5Then, the lightweight cross-domain matching algorithm of the present invention is used to match the two images with the dense model to solve the pose. According to the solved pose, a virtual AR object is registered in front of the landmark building in one image, and the virtual AR object is rendered on the other image, as shown in FIG. Figure 6 shown.

[0097] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for visually localizing a target building based on a generative dense model, characterized in that: include: Step 1: Collect an undistorted single-view image of the target building. Use the sparse multi-view generative diffusion model to expand the single-view image to generate images of the target building and its poses from multiple perspectives. The expanded image serves as the reference image for subsequent steps. Step 2: Based on all reference images, a 3D dense model is generated using the reconstructed large model based on sparse views; Step 3: Based on each reference image and its pose, render the rendered depth map with the same pose in the 3D dense model; Perform monocular depth estimation on the reference image to obtain an estimated depth map; Based on the rendered depth map and the estimated depth map, the scale ratio of the 3D dense model and the pose of each reference image are jointly optimized to obtain the optimized 3D dense model and the pose of each reference image; Step 4: Obtain a query image of the target building to be located, select the reference image that is most similar to the query image from all reference images, retrieve the optimized pose corresponding to the reference image, and render the selected reference image in the optimized 3D dense model based on the retrieved pose to obtain a reference depth map; Step 5: Feature extraction and 2D-2D cross-domain matching are performed on the query image and the filtered reference image to obtain matching feature point pairs. The feature points belonging to the reference image in the matching feature point pair are promoted to 3D using the reference depth map. The promoted feature points and the feature points belonging to the query image in the matching feature point pair form 2D-3D feature point pairs. The 2D-3D feature point pairs are used to solve the camera pose through the 6D pose solver to obtain the pose of the query image.

2. The target building visual positioning method based on generative dense model according to claim 1 is characterized in that: In step one, the sparse multi-view diffusion model is used to expand the single-view image to generate images of the target building and its poses from multiple perspectives. Specifically, the single-view image is input into the Zero-1-to-3 model, and 6 evenly distributed initial poses are set. The initial image corresponding to the initial pose is generated by the Zero-1-to-3 model, and the initial image is optimized and constrained by the ConsistNet module so that the 6 initial images meet the geometric consistency assumption. The optimized and constrained images and their corresponding initial poses are the images and poses of the target building from multiple perspectives.

3. The target building visual positioning method based on generative dense model according to claim 1 is characterized in that: In step 2, the large reconstructed model based on sparse views is a FS3D pre-trained model.

4. The target building visual positioning method based on a generative dense model according to claim 1 is characterized in that: In step three, the rendering in the 3D dense model is performed to obtain a rendered image and a depth map having the same posture, using a rasterization rendering method or a projection algorithm.

5. The target building visual positioning method based on generative dense model according to claim 1 is characterized in that: In step 3, the monocular depth estimation of the reference image is performed by processing the reference image using a Depth Anything model or a DPT model.

6. The target building visual positioning method based on generative dense model according to claim 1 is characterized in that: In step 3, the scale ratio of the 3D dense model and the pose of each reference image are jointly optimized based on the rendered depth map and the estimated depth map, specifically including: 1) According to the rendered depth map and the estimated depth map, the 2D pixel points of the area corresponding to the target building in the reference image are extracted, and the 2D pixel points are upscaled according to the rendered depth map and the estimated depth map to obtain two sets of 3D point clouds. in and is the 3D point obtained from the rendered depth map and the estimated depth map, Represents a three-dimensional point set; 2) The principal component analysis algorithm is used to calculate the eigenvalues ​​and eigenvectors corresponding to the first three principal components of the two groups of 3D point clouds respectively. The root mean square of the eigenvalues ​​obtained from the same group of 3D point clouds is solved as the scale coefficient of the 3D point cloud. The ratio of the scale coefficients of the two groups of 3D point clouds is calculated to obtain the prior scaling factor. The prior scaling factor is applied to the 3D point corresponding to the rendered depth map. On top, a set of pre-adjusted 3D point clouds is obtained; 3) Use the pose of the current reference image as the initial pose p; 4) Construct the three-dimensional scaling matrix S of the point cloud scale in the world coordinate system w =diag(s x , s y , s z ), where s x , s y , s z The x, y, and z scaling ratios of the point cloud are used to construct an optimization function E(p, S w ), iteratively optimize the pose p and three-dimensional scaling matrix S according to the optimization function w , until the convergence condition is reached and the optimized pose p is obtained * and the optimized 3D scaling matrix The optimization function E(p, S w )for: Among them, W ij is the weight, β is a hyperparameter, λ is a regularization weight, I is the identity matrix, and ||·|| is the L1 norm; 5) Repeat steps 1)-4) for each reference image to obtain multiple optimized poses p * and the 3D scaling matrix The geometric mean of the optimized three-dimensional scaling matrices is calculated element by element to obtain the final three-dimensional scaling matrix. According to the final three-dimensional scaling matrix The 3D dense model is scaled by the scaling ratios in the x, y, and z directions to obtain the optimized 3D dense model M * .

7. The target building visual positioning method based on generative dense model according to claim 6 is characterized in that: Step 4) The convergence condition is to optimize the function value E(p, S w ) is less than the preset convergence threshold ∈ or the number of iterations reaches 20 times.

8. The method for visually locating a target building based on a generative dense model according to claim 1, wherein: In step 5, feature extraction and 2D-2D cross-domain matching are performed on the query image and the selected reference image to obtain matching feature point pairs, which specifically includes: a) Using the visual basis model to analyze the query image I A and the filtered reference image I B Perform image encoding to obtain image visual feature maps b) Using lightweight feature matching to query image I A and reference image I B The feature points p of the query image and the reference image are extracted respectively A , p B , and calculate the descriptor d of the feature point A , d B ; c) For query image I A and reference image I B , calculate the feature similarity of every two feature points in the image in the corresponding image visual feature map; Among them, p i and p j For any feature point in the query image or reference image, F c (·) represents the visual feature vector of the feature point in the corresponding image visual feature map, Sim(p i , p j ) is the feature point p i and p j The feature similarity between d) Take the query image I A The feature points are nodes to establish an undirected feature point graph in the image Take reference image I B The feature points are nodes to establish an undirected feature point graph in the image Establish a directed feature point graph between images with the feature points of the query image and the feature points of the reference image as nodes and e) Based on the feature similarity between feature points, the self-attention mechanism and the cross-attention mechanism are used to propagate the edges in the undirected feature point graph within the image and the directed feature point graph between images, and the descriptors of the nodes are updated during the propagation process; based on the descriptors of the updated nodes, the query image I A The feature points and reference image I B Establish an association matrix between the feature points: in is the descriptor of the i-th feature point in the query image, is the descriptor of the jth feature point of the reference image, C i，j is the degree of association; f) The correlation degree is greater than the preset threshold ∈ c The query image I A The feature points and reference image I B The feature point combination is taken as the matching feature point pair.

9. The method for visually locating a target building based on a generative dense model according to claim 1, wherein: In step five, the 6D pose solver includes two parts: basic pose solution and bundle adjustment optimization; the 2D-3D feature point pairs are used to solve the camera pose through the 6D pose solver. Specifically, the 2D-3D feature point pairs are input into the basic pose solution part, which obtains the matching confidence of the 2D-3D feature point pairs, and solves the basic pose of the 2D-3D feature points using the RANSAC-based P3P algorithm under the guidance of the matching confidence; the obtained basic pose is optimized using the bundle adjustment method, and the optimized pose is the pose of the query image.

Images