Target building visual positioning method based on generative dense model

Abstract

The invention provides a target building visual positioning method based on a generative dense model. In order to solve the problem that a traditional visual positioning method seriously depends on an expensive three-dimensional reconstruction map, a two-stage positioning method is designed. In the off-line registration stage, a 3D dense model obtained by the generative model is used as input, a joint pose-scale optimization algorithm is used, the geometric proportion of the 3D model is optimized, and rendered images under multiple view angles and corresponding poses of the rendered images are registered; in the online positioning stage, based on the pre-trained lightweight cross-domain feature matcher and the robust pose solver, the camera pose corresponding to the query image can be quickly obtained. The method gives consideration to the positioning precision and speed, can be deployed and used in a strict environment with no map and less reference, and can remarkably improve the effect of a low-quality model in visual positioning. Compared with an existing visual positioning method, the method has the advantages that the applicability is wider, and the positioning result is more accurate.

Description

Technical Field

[0001] The present invention relates to the fields of augmented reality, visual positioning and generative visual models, and in particular to a method for visual positioning of a target building based on a generative dense model. Background Art

[0002] Augmented Reality (AR) is a new technology that integrates real-world information and virtual image information. It uses computer technology to simulate virtual information and then superimposes it on the real world, thereby achieving a new experience and application that goes beyond reality. With the development of computer vision technology, especially structure-from-motion mapping (SfM), simultaneous localization and mapping (SLAM), and visual localization technology, many AR applications for outdoor scenes have been born. Among them, visual localization is to calculate the camera position and posture of the captured image, that is, the camera pose, based on the relevant image information obtained by the camera or other visual sensors. The camera pose obtained based on visual localization can generate the corresponding AR effect.

[0003] Classic outdoor scene visual positioning solutions usually require a large number of images to be collected and calibrated for real scenes, and then a 3D point cloud map of the scene is constructed through offline or simultaneous mapping technology. In the positioning stage, the visual positioning algorithm or sensor information is used to infer the camera pose of the user shooting the scene at a certain location. However, due to the huge overhead of scene mapping and storage consumption, the deployment cost of this method is usually relatively high. For larger outdoor scenes or buildings, in order to obtain a relatively good positioning point cloud map, a large amount of high-quality data collection and calibration is usually required, and the length of the mapping will also increase exponentially with the scale that needs to be reconstructed. In order to solve the cost problem caused by mapping, visual positioning methods based on 3D dense models (such as grids) have been proposed, such as the deep learning accelerated visual positioning technology based on mesh representation proposed by Zhang P in the article "DLALoc: Deep-Learning AcceleratedVisual Localization Based on Mesh Representation". This method does not require complicated image acquisition and point cloud reconstruction, but is based on ready-made or low-precision grid models for positioning, which greatly saves mapping time and computing costs.

[0004] There are still some shortcomings in the existing visual positioning solutions based on 3D dense models. First, for the acquisition of 3D models, CAD models are generally downloaded from the Internet, or downloaded from open source vector map tools (such as Google Earth). This is an unstable resource acquisition channel. Once the required resources are not available, manual production or reconstruction is still required. Secondly, since most 3D dense models cannot be correctly aligned or scaled with the real scene, there are usually large errors in the geometric authenticity of the model, which will lead to the difficulty of converging to the correct result due to the huge 3D point error when positioning the query image. Third, since the appearance of the 3D dense model is different from the real scene, that is, there is a large error in the appearance authenticity of the model, the traditional local feature solution is difficult to match the model with a huge difference in feature distribution with the query image, or it needs to rely on a huge dense matching algorithm with extremely high overhead. The above problems are severe challenges to the availability, accuracy and efficiency of existing positioning algorithms.

[0005] In summary, traditional solutions have problems such as high mapping overhead, difficulty in obtaining available dense models, and low model quality leading to low positioning accuracy and efficiency. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a target building visual positioning method based on a generative dense model.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for visual positioning of a target building based on a generative dense model comprises the following steps:

[0009] Step 1: Collect a single-view image of the target building without distortion, and use the sparse multi-view generation diffusion model to expand the single-view image to generate images of the target building and its posture under multiple perspectives. The expanded image is used as a reference image for subsequent steps;

[0010] Step 2: Based on all reference images, a 3D dense model is generated using the reconstructed large model based on sparse views;

[0011] Step 3: Based on each reference image and its pose, render a rendered depth map with the same pose in the 3D dense model; perform monocular depth estimation on the reference image to obtain an estimated depth map; based on the rendered depth map and the estimated depth map, jointly optimize the scale ratio of the 3D dense model and the pose of each reference image to obtain the optimized 3D dense model and the pose of each reference image;

[0012] Step 4: Obtain a query image of the target building to be located, select a reference image that is most similar to the query image from all reference images, retrieve the optimized pose corresponding to the reference image, render the selected reference image in the optimized 3D dense model based on the retrieved pose, and obtain a reference depth map;

[0013] Step 5: Perform feature extraction and 2D-2D cross-domain matching on the query image and the screened reference image to obtain matching feature point pairs. The feature points belonging to the reference image in the matching feature point pair are promoted to 3D through the reference depth map. The promoted feature points and the feature points belonging to the query image in the matching feature point pair form 2D-3D feature point pairs. The 2D-3D feature point pairs are solved for camera pose by a 6D pose solver to obtain the pose of the query image.

[0014] Furthermore, in step one, the sparse multi-view generation diffusion model is used to expand the single-view image to generate images of the target building and its posture under multiple perspectives. Specifically, the single-view image is input into the Zero-1-to-3 model, 6 evenly distributed initial postures are set, and the initial image corresponding to the initial posture is generated by the Zero-1-to-3 model. The initial image is then optimized and constrained by the ConsistNet module so that the 6 initial images conform to the geometric consistency assumption. The optimized and constrained images and their corresponding initial postures are the images and postures of the target building under multiple perspectives.

[0015] Furthermore, in step 2, the large reconstructed model based on sparse views is a FS3D pre-trained model.

[0016] Furthermore, in step three, the rendering in the 3D dense model to obtain a rendered image and a depth map having the same posture is performed using a rasterization rendering method or a projection algorithm.

[0017] Furthermore, in step 3, the scale ratio of the 3D dense model and the pose of each reference image are jointly optimized based on the rendered depth map and the estimated depth map, specifically including:

[0018] 1) According to the rendered depth map and the estimated depth map, the 2D pixel points of the area corresponding to the target building in the reference image are extracted, and the 2D pixel points are upgraded according to the rendered depth map and the estimated depth map to obtain two sets of 3D point clouds. in and is the 3D point obtained from the rendered depth map and the estimated depth map, represents a three-dimensional point set;

[0019] 2) The principal component analysis algorithm is used to calculate the eigenvalues ​​and eigenvectors corresponding to the first three principal components of the two groups of 3D point clouds respectively. The root mean square of the eigenvalues ​​obtained from the same group of 3D point clouds is solved as the scale coefficient of the 3D point cloud. The ratio of the scale coefficients of the two groups of 3D point clouds is calculated to obtain the prior scaling factor. The prior scaling factor is applied to the 3D point corresponding to the rendered depth map. A set of pre-adjusted 3D point clouds is obtained;

[0020] 3) Use the pose of the current reference image as the initial pose p;

[0021] 4) Construct the three-dimensional scaling matrix S of the point cloud scale in the world coordinate system w =diag(s x ,s y ,s z ), where s x ,s y ,s z The x, y, and z scaling ratios of the point cloud are used to construct an optimization function E(p,S w ), iteratively optimize the pose p and the three-dimensional scaling matrix S according to the optimization function w , until the convergence condition is reached, the optimized pose p* and the optimized homogeneous two-dimensional scaling matrix S are obtained w * ; The optimization function E(p,S w )for:

[0022]

[0023] Among them, W ij is the weight, β is a hyperparameter, λ is a regularization weight, I is the identity matrix, and ‖·‖ is the L1 norm;

[0024] 5) Repeat steps 1)-4) for each reference image to obtain multiple optimized poses p * And the 3D scaling matrix The geometric mean of the optimized three-dimensional scaling matrices is calculated element by element to obtain the final three-dimensional scaling matrix. According to the final three-dimensional scaling matrix The 3D dense model is scaled according to the scaling ratios in the x, y, and z directions to obtain the optimized 3D dense model M * .

[0025] Furthermore, in step 5, the feature extraction and 2D-2D cross-domain matching of the query image and the selected reference image to obtain matching feature point pairs specifically includes:

[0026] a) Using the visual base model to analyze the query image IA and the selected reference image I B Perform image encoding to obtain image visual feature maps

[0027] b) Using lightweight feature matching to query image I A and reference image I B The feature points p of the query image and the reference image are extracted respectively A ,p B , and calculate the descriptor d of the feature point A ,d B ;

[0028] c) For the query image I A and reference image I B , calculate the feature similarity of every two feature points in the image in the corresponding image visual feature map;

[0029]

[0030] Among them, p i and p j For any feature point in the query image or reference image, F c (·) represents the visual feature vector of the feature point in the corresponding image visual feature map, Sim(p i ,p j ) is the feature point p i and p j The feature similarity between them;

[0031] d) Take the query image I A The feature points are used as nodes to establish an undirected feature point graph in the image Take the reference image I B The feature points are used as nodes to establish an undirected feature point graph in the image Establish a directed feature point graph between images with the query image feature points and the reference image feature points as nodes and

[0032] e) Based on the feature similarity between feature points, the self-attention mechanism and the cross-attention mechanism are used to propagate the edges in the undirected feature point graph within the image and the directed feature point graph between images, and the descriptors of the nodes are updated during the propagation process; based on the descriptors of the updated nodes, the query image I A The feature points and reference image I B Establish a correlation matrix between the feature points:

[0033]

[0034] in is the descriptor of the i-th feature point in the query image, is the descriptor of the jth feature point of the reference image, C i,j is the degree of relevance;

[0035] f) The correlation degree is greater than the preset threshold ∈ c The query image I A The feature points and reference image I B The feature point combination is taken as the matching feature point pair.

[0036] Furthermore, in step five, the 6D pose solver includes two parts: basic pose solution and bundle adjustment optimization; the 2D-3D feature point pairs are used to solve the camera pose through the 6D pose solver as follows: the 2D-3D feature point pairs are input into the basic pose solution part, which obtains the matching confidence of the 2D-3D feature point pairs, and solves the basic pose of the 2D-3D feature points using the RANSAC-based P3P algorithm under the guidance of the matching confidence; the obtained basic pose is optimized using the bundle adjustment method, and the optimized pose is the pose of the query image.

[0037] The beneficial effects of the present invention are:

[0038] 1) Regarding the acquisition of 3D dense models, this invention uses advanced 3D asset generation technology, takes sparse perspective reconstruction of large models as the theoretical basis and foundation, and combines robust multi-perspective image generation to quickly obtain basic 3D dense models as positioning assets. The generated 3D dense model solves the map-building dependency of visual positioning and the problem of missing 3D model sources, and is easy to deploy;

[0039] 2) Regarding the quality of 3D dense models, the present invention optimizes from two perspectives. First, during generation, the supervision of multi-view images is used to ensure the degree of model restoration as much as possible. Secondly, the present invention considers using sparse view reference images combined with model parameters for joint optimization, and the joint pose-scale optimization method uses an iterative optimization strategy to simultaneously optimize the ratio of 2D images and 3D models, solve the problem of poor collective authenticity of 3D dense models, and effectively obtain usable map data in visual positioning;

[0040] 3) Regarding the robustness of online positioning, the present invention uses an efficient local feature matching algorithm guided by a visual base model and can be applied to cross-domain image matching to quickly obtain a robust correspondence between a real query image and a generated reference image, thereby performing high-precision pose estimation. The cross-domain local feature matching algorithm can perform cross-domain matching, so there is no need to render the 3D dense model during the online positioning stage, and there is no need to obtain a real pre-annotated building image as a positioning map database image, which improves the efficiency and robustness of positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the overall framework diagram of the present invention;

[0042] Figure 2 is a schematic diagram of a cross-domain matching network in the present invention;

[0043] Figure 3 is a line graph of the experimental accuracy of the cross-domain matching network in the present invention;

[0044] Figure 4 It is a time-precision scatter plot of experimental comparison between the present invention and other methods;

[0045] Figure 5 It is a 3D dense model reconstruction rendering of the target building of the present invention;

[0046] Figure 6 This is an effect diagram of the application of the present invention in AR technology. DETAILED DESCRIPTION

[0047] The present invention is a target building visual positioning technology based on a generative dense model. In the offline stage, a single or a small number of target building images are input to generate a 3D dense model, generate a reference image and optimize the model. In the online stage, a query image and internal parameters are input to output the corresponding camera 6-DOF posture.

[0048] This method is implemented based on Python and C++. Figure 1 As shown in the figure, this method can be divided into an optimization phase and a positioning phase. The optimization phase needs to be deployed and processed on a device with GPU computing resources, while the positioning phase has lower requirements on the GPU computing power and video memory. The optimization phase can be divided into three modules: the 2D generation module corresponds to step 1, the 3D generation module corresponds to step 2, and the optimization registration module corresponds to step 3; the online phase can be divided into three modules: the feature extraction module and the cross-domain matching module correspond to step 4, and the pose solution module provides the final output of the method.

[0049] The present invention is described in detail below with reference to the accompanying drawings. The technical features of each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0050] The main steps of the method of the present invention are as follows:

[0051] Step 1: First, collect images of the target building from a relatively complete viewing angle. The collected single-view image without distortion is recorded as Then, the diffusion model F is generated using sparse multi-view I , generate a set of multi-view images of several target buildings And the corresponding posture Where N is the number of viewing angles, usually N = 6;

[0052] The sparse view generative diffusion model is a pre-trained model that accepts an image I0 at a single view as input, and independently and individually models the conditional edge distribution of the image to generate a new set of images of 6 views with uniform view layout. To ensure the geometric and appearance consistency of the generated image. The model uses linear noise scheduling to further improve the image consistency between multiple views. The model uses two-stage training and scaled reference attention mechanism to further improve the model's performance in the semantic content and texture details of the input image in the generated image. In order to adapt to the application scenario, the model is fine-tuned in the Warehouse public model dataset. Collect building models from the Warehouse dataset, adjust them to the unit cube of [-0.5, 0.5], and uniformly sample the polar angle θ and azimuth angle from the spherical coordinate system according to the following formula And the radial distance r generates the rendering perspective, and the rendered image is used as the supervision truth value:

[0053]

[0054] In the specific implementation, the sparse multi-view diffusion network F IUse the Zero-1-to-3 model (Liu R, Wu R, Van Hoorick B, Tokmakov P, Zakharov S, Vondrick C. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE / CVF international conference on computer vision 2023 (pp. 9298-9309).) to pre-train the network. Input image I0 into the Zero-1-to-3 model, and use the model to generate 6 initial images under uniformly arranged viewing angles. After the initial image is generated, the 6 views should meet the geometric consistency assumption, that is, each pixel should be consistent when back-projected into the feature volume. According to this principle, the ConsistNet module (Yang J, Cheng Z, Duan Y, Ji P, Li H. Consistnet: Enforcing 3d consistency for multi-view images diffusion. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 7079-7088).) is connected after the Zero-1-to-3 model to optimize constraints. The ConsistNet module first projects the view onto the global 3D voxel grid and infers the consistency, then samples the light in the global 3D voxel to obtain consistent 3D features and aggregate them into each view to supervise consistency.

[0055] Step 2: Obtain a multi-view image set Then, the large model F is reconstructed based on sparse views. M , generate a 3D dense model (3D mesh or dense point cloud) M;

[0056] The sparse view-based reconstruction model is a pre-trained model. The model accepts a multi-view image set Based on the transformer backbone network, the image is mapped into an implicit triplane structure with multi-view supervision, and a dense 3D mesh model M is generated by FlexiCube or Splatting. The training of this model is based on the visual transformer encoder and the triplane encoder, and is supervised by two stages from implicit training to mesh training. In the specific implementation, the reconstructed large model F based on the sparse view MThe FS3D (Zhu J, Ma H, Chen J, et al. Few-shot 3D shape generation [J]. arXiv preprint arXiv: 2305.11664, 2023.) pre-trained model was used.

[0057] Step 3: Get an image collection Initial pose set After obtaining the dense 3D model M, a pose-scale joint optimization algorithm is used to simultaneously optimize the image pose and the scale ratio of the model, so that the fit between the image and the model and the geometric accuracy of the model are more accurate, and the optimized pose set and the 3D model after non-rigid scaling are obtained;

[0058] The pose-scale joint optimization algorithm described in step 3 is the core of the present invention, and the specific steps are as follows:

[0059] (3.1) For each input reference image I and the corresponding pose p, the corresponding rendered depth map is rendered in the 3D grid M The rendering method here can use simple rasterization rendering, or use a projection algorithm;

[0060] (3.2) For each input reference image I, use the monocular depth estimation model to obtain the corresponding estimated depth map

[0061] (3.3) Through the two depth maps, the target building parts in the image are lifted respectively to obtain two sets of 3D point clouds in and 3D points obtained for rendering depth and estimating depth, Represented as a three-dimensional point set;

[0062] (3.4) Perform principal component analysis (PCA) on the two groups of point clouds, decompose the eigenvalues ​​and eigenvectors corresponding to the first three principal components, solve the root mean square of the eigenvalues ​​as the scale coefficient of the point cloud, calculate the ratio to obtain the prior scaling factor of the two groups of points, and apply it to the three-dimensional points corresponding to the rendering depth on, so that it is The scale difference is within acceptable range; apply a priori scaling factor to 3D points The above refers to the prior scaling factor and the three-dimensional point Multiply to update the 3D point

[0063] (3.5) Construct an optimization function, use least squares optimization to optimize the function, and add the three-dimensional scaling matrix of the point cloud scale in the world coordinate system. The world coordinate system is the coordinate system where the dense model is located, and the three-dimensional scaling matrix is ​​the diagonal matrix S w =diag(s x ,s y ,s z ), where s x ,s y ,s z is the x, y, and z scale ratio of the model in the world coordinate system. Based on the optimization function, iteratively optimize the pose p and the scaling matrix until the convergence condition is reached. The optimization objects are as follows:

[0064]

[0065] Among them, W ij is the weight, which is determined according to the distance between points. β is a hyperparameter used to control the influence of the squared distance term on the weight, λ is the regularization weight, I is the identity matrix, and ‖·‖ is the L1 norm;

[0066] (3.6) Repeat steps 1)-4) for each reference image to obtain multiple optimization results p * , And calculate the geometric mean of the optimized multiple three-dimensional scaling matrices element by element to obtain the final three-dimensional scaling matrix The final three-dimensional scaling matrix Applied to the 3D dense model M, the optimized model M is obtained * The scaling ratio S w Applied to the 3D dense model M, the specific method is to transform all vertex coordinates of the 3D dense model with the scaling factor S w Multiply.

[0067] In a specific implementation, step (3.1) can render the 3D mesh M using a rasterizer based on OpenGL to obtain fast rendering results, or it can pursue quality and use a rendering engine such as Blender to obtain a rendering result with a strong sense of reality. Since rendering only exists in the optimization stage, the latter can be selected to improve the quality of feature points extracted in subsequent steps.

[0068] In the specific implementation, the specific implementation method of step (3.2) for monocular depth estimation of the image is to use the monocular depth estimation model Depth Anything (Yang L, Kang B, Huang Z, et al. Depth anything: Unleashing the power of large-scale unlabeled data [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024: 10371-10381.). This model accepts a single image as input and directly outputs a relative depth image. The DPT model (dense prediction transformer) can also be used.

[0069] The convergence condition of step (3.5) in the joint optimization algorithm is as follows: set a convergence threshold ∈, when E(p,S w )<∈, the model and image at this perspective are considered to have been well aligned and fitted, and the optimization is completed. Or when the number of iterations reaches 20, the algorithm is considered to have reached a local optimum and the optimization is stopped.

[0070] In the optimization object of step (3.5) of the joint optimization algorithm, the regularization parameter λ is generally set to 1.0. When the initial model is far from the actual target building, the regularization parameter λ can be reduced to 0.1. In order to obtain automatic adaptation, a smaller regularization parameter is usually used in the first round of iteration, and a larger regularization parameter is used in subsequent iterations.

[0071] In the optimization object of step (3.5) of the joint optimization algorithm, the weight hyperparameter β is generally set to 0.1. When the initial model is far from the actual target building, the weight hyperparameter β can be reduced to 0.01. In order to obtain automatic adaptation, a smaller weight hyperparameter is usually used in the first round of iteration, and a larger weight hyperparameter is used in subsequent iterations.

[0072] Step 4: Obtain a query image of the target building to be located, select the reference image most similar to the query image from all reference images, retrieve the optimized pose and 3D dense model corresponding to the reference image, render the filtered reference image in the retrieved 3D dense model based on the retrieved pose, and obtain a reference depth map.

[0073] In step 4, the selection of the reference image most similar to the query image is implemented by a global feature module, such as the MixVPR algorithm (A.Ali-Bey, B.Chaib-Draa and P.Giguére, "MixVPR: Feature Mixing for Visual Place Recognition," 2023IEEE / CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2023, pp.2997-3006, doi: 10.1109 / WACV56688.2023.00301.). Specifically, the global feature module is used to extract the global features of each reference image and the query image, calculate the feature vector cosine similarity between the global features of each reference image and the global features of the query image, and select the reference image with the highest similarity.

[0074] Step 5: In the online positioning stage, a lightweight cross-domain feature matching algorithm is used to extract features and perform 2D-2D cross-domain matching on the query image and the most similar reference image, and then it is promoted to 3D through the reference depth map. Then, the camera pose is solved through a robust 6D pose solver (PnP algorithm and bundle adjustment optimization).

[0075] The lightweight cross-domain feature matching algorithm described in step 5 is implemented by a pre-trained model that accepts two images I with significantly different statistical distributions (e.g., synthetic images and real images). A ,I B , perform coarse matching through the DINOv2 base model, and perform fine matching through the cross attention mechanism, and finally output two sets of corresponding feature point coordinates The algorithm is built as a two-stage approach, such as Figure 2 As shown, the details are as follows:

[0076] 5.1) The query image I A and the reference image I with the highest similarity B Input, for the input image I A ,I B , using the DINOv2 model for image encoding to obtain a robust coarse-precision level image visual feature map

[0077] 5.2) For image I A ,I B The feature points p of the image are extracted using the XFeat algorithm A ,p B , and calculate the descriptor d of the feature pointA ,d B , obtain a set of feature points and descriptors of coarse precision level {d A ,p A},{d B ,p B};

[0078] 5.3) Using visual features The feature similarity matrix is ​​established for every two feature points and descriptors, taking the query image to the reference image as an example:

[0079]

[0080] in, represents the visual feature vector at a certain position, is the i-th feature point of the query image, is the jth feature point of the query image, For feature points and The feature similarity between .

[0081] 5.4) To query the feature point p of the image A As nodes, connect any two nodes to establish an undirected feature point graph in the image Take the feature point p of the reference image B As nodes, connect any two nodes to establish an undirected feature point graph in the image Take the query image feature points and the reference image feature points as nodes and connect the query image node p A and the node p of the reference image B , establish a directed feature point graph between images and picture The direction of the middle edge is from p A to p B ,picture The direction of the middle edge is from p B to p A ; The weight of the edge connecting two nodes in the feature point graph is the feature similarity between the two nodes.

[0082] 5.5) For undirected feature point graphs and directed feature point graphs, the self-attention mechanism based on position encoding propagates the edges in the undirected feature point graph within the image, and the cross-attention mechanism propagates the edges in the directed feature point graph between images, and updates the descriptor features of the nodes during the propagation process. Specifically, each node in the graph receives the descriptor features of the adjacent nodes, and calculates the attention weight based on the feature similarity as the weight, which is used as the new node feature.

[0083] The above propagation process is repeated three times to obtain the optimized feature point graph and the node descriptor features finally updated in the feature point graph.

[0084] 5.6) Directed feature point graph between optimized images Updated node descriptor features of feature points corresponding to the common part To establish the correlation matrix:

[0085]

[0086] Retain the correlation matrix with a correlation greater than the threshold ∈ c The point pairs are filtered and the corresponding relationship of the feature points is used as the matching feature point pairs, and the corresponding feature point association degree in the association matrix is ​​output as the matching confidence.

[0087] Furthermore, the feature points belonging to the reference image in the updated matching feature point pair are lifted to 3D by referring to the depth map, and the lifted feature points and the feature points belonging to the query image in the original matching feature point pair form a 2D-3D feature point pair.

[0088] Furthermore, the robust 6D pose solver described in step five includes two parts: RANSAC sampling based on matching confidence prior and basic pose solution of PnP algorithm, and bundle adjustment optimization based on reprojection error. Specifically, the 2D-3D feature point pairs are input into the first part, RANSAC sampling is performed based on the matching confidence prior of the feature point pairs, and the sampled samples are solved by the PnP algorithm to obtain the basic pose; the solved basic pose is optimized by bundle adjustment in the second part, and the optimized pose is the pose of the query image finally obtained. The matching confidence corresponds to the Gaussian correlation score of the feature point pairs obtained before. When solving, different matching pairs will be weighted according to the confidence, thereby guiding the sampling of RANSAC.

[0089] In the specific implementation of the lightweight cross-domain feature matching algorithm described in step 5, the DINOv2 (ViT-S) model was selected to extract robust coarse-precision visual features of the image to balance time and accuracy.

[0090] In order to further demonstrate the implementation effect of the present invention, the present invention is compared with previous positioning solutions and processes in multiple aspects, two experiments are conducted, and the actual AR effect is demonstrated.

[0091] The present invention is compared with the baseline solution on the CADLoc dataset. The CADLoc dataset contains landmark buildings of four classic tourist attractions (Notre Dame de Paris, Pantheon, Reichstag, St. Peter's Square). Each building contains 3D mesh models of different qualities for positioning. The model quality includes high-quality multi-view geometry reconstruction results, as well as low-quality models manually reconstructed by artists, and even CAD models without textures. In order to test the positioning effect of the present invention on the generative model, a 3D mesh obtained based on the generative model was added to each scene in the experiment for testing. The experimental environment is Ubuntu 18.04, and the experiment was carried out under a GTX1080 graphics card (12G video memory). In the offline registration stage, the maximum video memory consumption is about 8G (dense matching RoMa peak requires 12G), and the video memory consumption in the online positioning stage is generally less, requiring only less than 4G, and can even be completely based on the CPU.

[0092] Experiment 1: Comparison of positioning accuracy and success rate

[0093] Figure 3 The positioning reprojection error recall rates of the positioning method of the present invention and the existing grid-based visual positioning method (MeshLoc) for four 3D models of different qualities (corresponding to A, B, C, and D) in the same landmark scene are demonstrated. The closer the curve is to the upper side, the higher the recall rate is under the same error threshold and the better the effect is. The present invention has significant improvements in both the absolute accuracy of positioning and the recall rate of dense reprojection error (Dense Re-Projection Error, DCRE), especially in the positioning of low-quality models (C and D). In the results of the generative model, the positioning results of this scheme are also significantly better than those of the baseline scheme, and are close to the positioning results reconstructed by MVS (multi-view geometry). This experiment proves the feasibility of this scheme and the guarantee of AR effect.

[0094] Experiment 2: Comparison of online positioning efficiency

[0095] Figure 4 The speed and accuracy of this solution and the baseline solution under different feature matching schemes are compared in the online positioning stage. The horizontal axis represents the number of image pairs processed per second, and the vertical axis represents the matching accuracy. Due to the reduction of dense feature dependency and rendering and retrieval time, the operating efficiency of this solution is significantly improved. This experiment proves the efficiency of this solution in practical applications.

[0096] The practical effects of the present invention are as follows Figure 5 and Figure 6 As shown: Through the dense model generation and joint optimization method of the present invention, a 3D dense model of the landmark building is reconstructed from a single-view image, such as Figure 5Then, the lightweight cross-domain matching algorithm of the present invention is used to match the two images with the dense model to solve the pose. According to the solved pose, a virtual AR object is registered in front of the landmark building in one image, and the virtual AR object is rendered on the other image, as shown in FIG. Figure 6 shown.

[0097] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for visual positioning of target buildings based on a generative dense model, characterized in that: include: Step 1: Collect a single-view image of the target building without distortion, and use the sparse multi-view generation diffusion model to expand the single-view image to generate images of the target building and its posture under multiple perspectives. The expanded image is used as a reference image for subsequent steps; Step 2: Based on all reference images, a 3D dense model is generated using the reconstructed large model based on sparse views; Step 3: Based on each reference image and its pose, render the rendered depth map with the same pose in the 3D dense model; Perform monocular depth estimation on the reference image to obtain an estimated depth map; Based on the rendered depth map and the estimated depth map, the scale ratio of the 3D dense model and the pose of each reference image are jointly optimized to obtain the optimized 3D dense model and the pose of each reference image; Step 4: Obtain a query image of the target building to be located, select a reference image that is most similar to the query image from all reference images, retrieve the optimized pose corresponding to the reference image, render the selected reference image in the optimized 3D dense model based on the retrieved pose, and obtain a reference depth map; Step 5: Perform feature extraction and 2D-2D cross-domain matching on the query image and the screened reference image to obtain matching feature point pairs. The feature points belonging to the reference image in the matching feature point pair are promoted to 3D through the reference depth map. The promoted feature points and the feature points belonging to the query image in the matching feature point pair form 2D-3D feature point pairs. The 2D-3D feature point pairs are solved for camera pose by a 6D pose solver to obtain the pose of the query image.

2. The target building visual positioning method based on the generative dense model according to claim 1 is characterized in that: In step one, the sparse multi-view generation diffusion model is used to expand the single-view image to generate images of the target building and its posture under multiple perspectives. Specifically, the single-view image is input into the Zero-1-to-3 model, 6 evenly distributed initial postures are set, and the initial image corresponding to the initial posture is generated by the Zero-1-to-3 model. The initial image is then optimized and constrained by the ConsistNet module so that the 6 initial images meet the geometric consistency assumption. The optimized and constrained images and their corresponding initial postures are the images and postures of the target building under multiple perspectives.

3. The target building visual positioning method based on a generative dense model according to claim 1, characterized in that: In step 2, the large reconstructed model based on sparse views is a FS3D pre-trained model.

4. The target building visual positioning method based on a generative dense model according to claim 1, characterized in that: In step three, the rendering in the 3D dense model obtains a rendered image and a depth map having the same posture, using a rasterization rendering method or a projection algorithm.

5. The method for visually locating a target building based on a generative dense model according to claim 1, characterized in that: In step 3, the monocular depth estimation of the reference image is performed by processing the reference image using a Depth Anything model or a DPT model.

6. The target building visual positioning method based on a generative dense model according to claim 1, characterized in that: In step 3, the scale ratio of the 3D dense model and the pose of each reference image are jointly optimized based on the rendered depth map and the estimated depth map, specifically including: 1) According to the rendered depth map and the estimated depth map, the 2D pixel points of the area corresponding to the target building in the reference image are extracted, and the 2D pixel points are upgraded according to the rendered depth map and the estimated depth map to obtain two sets of 3D point clouds. in and is the 3D point obtained from the rendered depth map and the estimated depth map, represents a three-dimensional point set; 2) The principal component analysis algorithm is used to calculate the eigenvalues ​​and eigenvectors corresponding to the first three principal components of the two groups of 3D point clouds respectively. The root mean square of the eigenvalues ​​obtained from the same group of 3D point clouds is solved as the scale coefficient of the 3D point cloud. The ratio of the scale coefficients of the two groups of 3D point clouds is calculated to obtain the prior scaling factor. The prior scaling factor is applied to the 3D point corresponding to the rendered depth map. A set of pre-adjusted 3D point clouds is obtained; 3) Use the pose of the current reference image as the initial pose p; 4) Construct the three-dimensional scaling matrix S of the point cloud scale in the world coordinate system w =diag(s x ,s y ,s z ), where s x ,s y ,s z The x, y, and z scaling ratios of the point cloud are used to construct an optimization function E(p, S w ), iteratively optimize the pose p and the three-dimensional scaling matrix S according to the optimization function w , until the convergence condition is reached, and the optimized posture p is obtained * And the optimized 3D scaling matrix The optimization function E(p, S w )for: Among them, W ij is the weight, β is a hyperparameter, λ is a regularization weight, I is the identity matrix, and ||·|| is the L1 norm; 5) Repeat steps 1)-4) for each reference image to obtain multiple optimized poses p * And the 3D scaling matrix The geometric mean of the optimized three-dimensional scaling matrices is calculated element by element to obtain the final three-dimensional scaling matrix. According to the final three-dimensional scaling matrix The 3D dense model is scaled according to the scaling ratios in the x, y, and z directions to obtain the optimized 3D dense model M * .

7. The target building visual positioning method based on a generative dense model according to claim 6, characterized in that: Step 4) The convergence condition is to optimize the function value E(p, S w ) is less than the preset convergence threshold ∈ or the number of iterations reaches 20 times.

8. The target building visual positioning method based on a generative dense model according to claim 1, characterized in that: In step 5, the feature extraction and 2D-2D cross-domain matching of the query image and the selected reference image are performed to obtain matching feature point pairs, which specifically includes: a) Using the visual base model to analyze the query image I A and the selected reference image I B Perform image encoding to obtain image visual feature maps b) Using lightweight feature matching to query image I A and reference image I B The feature points p of the query image and the reference image are extracted respectively A , p B , and calculate the descriptor d of the feature point A , d B ; c) For the query image I A and reference image I B , calculate the feature similarity of every two feature points in the image in the corresponding image visual feature map; Among them, p i and p j For any feature point in the query image or reference image, F c (·) represents the visual feature vector of the feature point in the corresponding image visual feature map, Sim(p i , p j ) is the feature point p i and p j The feature similarity between them; d) Take the query image I A The feature points are used as nodes to establish an undirected feature point graph in the image Take the reference image I B The feature points are used as nodes to establish an undirected feature point graph in the image Establish a directed feature point graph between images with the query image feature points and the reference image feature points as nodes and e) Based on the feature similarity between feature points, the self-attention mechanism and the cross-attention mechanism are used to propagate the edges in the undirected feature point graph within the image and the directed feature point graph between images, and the descriptors of the nodes are updated during the propagation process; based on the descriptors of the updated nodes, the query image I A The feature points and reference image I B Establish a correlation matrix between the feature points: in is the descriptor of the i-th feature point in the query image, is the descriptor of the jth feature point of the reference image, C i，j is the degree of relevance; f) The correlation degree is greater than the preset threshold ∈ c The query image I A The feature points and reference image I B The feature point combination is taken as the matching feature point pair.

9. The method for visual positioning of target buildings based on a generative dense model according to claim 1, characterized in that: In step five, the 6D pose solver includes two parts: basic pose solving and bundle adjustment optimization; the 2D-3D feature point pairs are used to solve the camera pose through the 6D pose solver. Specifically, the 2D-3D feature point pairs are input into the basic pose solving part, which obtains the matching confidence of the 2D-3D feature point pairs, and solves the basic pose of the 2D-3D feature points using the RANSAC-based P3P algorithm under the guidance of the matching confidence; the obtained basic pose is optimized using the bundle adjustment method, and the optimized pose is the pose of the query image.

Images