A 3D Gaussian visual localization method for sparse view

Abstract

The present invention discloses a three-dimensional Gaussian visual positioning method for sparse perspective, which belongs to the field of positioning technology and is used for visual positioning. The method comprises extracting global image features and local image features of a sparse image and constructing a feature database; performing nearest neighbor search on the feature database and the RGB image to obtain a sparsity index, calculating the camera pose of a pseudo-perspective, rendering a color map and a depth map under the pseudo-perspective using a trained main model, extracting global features and local features and incorporating them into the feature database, constructing a 2D-3D correspondence relationship in combination with the target color map, performing dynamic inlier screening and pose solving, and obtaining a positioning result. The present invention intelligently supplements the perspective loss of key areas, effectively improves the balance and coverage of feature expression in three-dimensional scenes, realizes dynamic updating of 2D-3D matching relationships and elimination of false matches, thereby significantly improving positioning accuracy and system stability, and having stronger robustness and practical usability under sparse training conditions.

Description

Technical Field

[0001] The invention discloses a three-dimensional Gaussian vision positioning method for sparse viewing angles, belonging to the technical field of positioning. Background Art

[0002] With the growing demand for high-precision 3D perception and pose estimation in scenarios such as augmented reality (AR), robot navigation, and autonomous driving, visual localization technology, as its core support, has received widespread attention. Current mainstream methods include deep learning-based absolute pose regression (APR), scene coordinate regression (SCR), and the rapidly developing 3D neural rendering method (NeRF). The APR method uses a neural network to predict the camera pose of an image end-to-end, avoiding explicit geometric modeling. The SCR method uses algorithms such as PnP to estimate the camera pose by leveraging the regression relationship between image pixels and 3D spatial points. Furthermore, neural rendering methods such as NeRF establish continuous volume density and color fields to model the implicit 3D structure of a scene. Some studies have attempted to incorporate these methods into visual localization tasks to assist in recovering geometric information.

[0003] However, the above methods still face a series of challenges in actual deployment. They generally rely on a large number of densely collected training images to support the model's spatial structure modeling and feature generalization capabilities. When the image distribution is sparse and the view coverage is uneven, it is difficult for the model to capture the true geometric features of the scene, resulting in reduced matching accuracy and unstable pose estimation. Especially under sparse view angles, the overlapping area between images is significantly reduced, which seriously restricts the effective matching between 2D key points and 3D points, resulting in an increase in the proportion of mismatches and an increase in pose errors. At the same time, neural rendering methods generally have high computational overhead, slow convergence speed, and are highly dependent on view density, making it difficult to meet the efficiency and response speed requirements of mobile terminals or real-time systems.

[0004] Therefore, there is an urgent need for a highly robust visual localization method that can adapt to sparse viewing conditions, build stable and accurate three-dimensional structures under conditions of limited training samples and weak geometric constraints, and improve the reliability of feature matching and pose estimation to better serve practical application needs. Summary of the Invention

[0005] The purpose of the present invention is to provide a three-dimensional Gaussian vision positioning method for sparse perspectives, so as to solve the problem that the existing technology is difficult to construct a three-dimensional structure with strong geometric consistency under the conditions of sparse training images and insufficient geometric constraints, resulting in unstable feature matching, low positioning accuracy or even inability to complete positioning.

[0006] A 3D Gaussian visual localization method for sparse viewpoints, including training a 3D Gaussian sputtering model using RGB images and camera poses, generating color images and depth maps through 2D projection and differentiable rendering mechanisms, extracting global and local image features from the sparse image, and building a feature database.

[0007] The feature database and RGB image are searched for the nearest neighbor to obtain the sparsity index, and the camera pose of the pseudo-perspective is calculated. The trained main model is used to render the color image and depth image under the pseudo-perspective. Global and local features are extracted and incorporated into the feature database. The 2D-3D correspondence relationship is constructed in combination with the target color image. Dynamic inlier screening and pose solution are performed to obtain the positioning result.

[0008] Training the three-dimensional Gaussian sputtering model includes: the RGB image is a sparse perspective image, the same sparse perspective image is used as the input of the three-dimensional Gaussian sputtering model, and the three-dimensional Gaussian sputtering model is constructed by a dual-branch three-dimensional Gaussian point cloud model collaborative modeling method, including training the main model and auxiliary supervision models , each three-dimensional Gaussian point cloud model includes multiple Gaussian points, and the Gaussian points are quadruples , 、 、 、 are the spatial position, the covariance matrix of the spatial scale and orientation of the Gaussian point, the color value and the opacity;

[0009] The same sparse view images are fed into the training main model and auxiliary supervision models , constructing cross-model view based on the rendered image under the same pseudo-perspective Figure 1 Consistency loss, loss function for:

[0010] ;

[0011] ;

[0012] Where, is the balance coefficient, and They are and Rendered image at the same pseudo perspective, represents the L1 norm, represents the structural similarity index;

[0013] Color reconstruction loss for a single 3D Gaussian point cloud model for:

[0014] ;

[0015] Where, is the pixel color of the training image, is the corresponding pixel value of the rendered image, Pixels , is the pixel color;

[0016] Total loss function for dual-branch training for:

[0017] ;

[0018] Where, is the weight coefficient, which is used to adjust the cross-model view Figure 1 The proportion of fatal losses in total losses;

[0019] The training of the 3D Gaussian sputtering model is completed by minimizing the total loss of the two-branch training.

[0020] Generating color images and depth maps involves projecting the three-dimensional covariance matrix into a two-dimensional covariance matrix based on the Jacobian transformation at the camera's perspective. :

[0021] ;

[0022] Where 𝐽 is the Jacobian matrix of the affine approximation of the projection function, is the view transformation matrix of the camera's viewing direction;

[0023] For each pixel in the image plane ,according to Sort the two-dimensional distances from all visible Gaussian points and construct a Gaussian set , using alpha-blending rendering to accumulate color values, generating ,Will As a color image:

[0024] ;

[0025] ;

[0026] Where, Gaussian point The cumulative transmittance, Gaussian point In pixels The shadow opacity, Indicates the Gaussian points, Indicates the Gaussian points, Gaussian point In pixels The shadow opacity;

[0027] Generate a depth map, the depth of a single pixel for:

[0028] ;

[0029] Where, Represents Gaussian points The absolute depth value at the current viewing angle will be As a depth map.

[0030] Constructing a feature database includes constructing a global image feature set and a local image feature set;

[0031] The global image feature set for:

[0032] ;

[0033] Where, is the first global eigenvectors, is the first A rotation matrix, is the first translation vector, yes The maximum value of

[0034] The local image feature set for:

[0035] ;

[0036] Where, is the three-dimensional space coordinate obtained by back-projection of local key points of the image, For Bound local feature descriptors.

[0037] Constructing a global image feature set includes extracting two-dimensional key point coordinates from sparse view images using an image local feature extraction network and corresponding , combined with the camera intrinsic parameter matrix corresponding to the sparse view image , external parameter matrix , the depth value of each key point in the depth map , map the two-dimensional key points to three-dimensional space and calculate the corresponding :

[0038] ;

[0039] Where, is the camera rotation matrix in the extrinsic matrix, is the translation vector in the external parameter matrix, Corresponding Binding to form local three-dimensional feature items And stored in the local image feature set.

[0040] Obtaining the sparsity index includes: Gaussian points in the three-dimensional Gaussian sputtering model ,statistics The visible perspective set , Equivalent to all observable Sparse view images, for the Sparse view images ,set up The set of observed Gaussian points is , The sparsity index for:

[0041] ;

[0042] Two images and The visual overlap of the two visible Gaussian point sets is the intersection ratio of the two visible Gaussian point sets. :

[0043] ;

[0044] Where, It is A sparse perspective image, yes The set of observed Gaussian points;

[0045] Using sparsity metrics Build a sparsity-sorted list.

[0046] Calculating the pseudo-perspective camera pose involves selecting the lowest sparsity from the sparsity sorted list As the target perspective to be enhanced, the one with the same of Greater than the set threshold Image collection :

[0047] ;

[0048] For The reference image with the farthest Euclidean distance in space, The posture is , is the translation vector, The rotation angle represented by the unit quaternion;

[0049] set up The camera pose is , the pseudo-view pose generated by interpolation is defined as:

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] Where, is the pseudo-perspective translation vector, is the pseudo-view rotation angle, is the interpolation factor, is the intermediate parameter.

[0055] Rendering color images and depth maps in pseudo-perspective using the trained main model includes using the same method as generating color images and depth maps, using the trained main model Camera pose in pseudo-perspective Generate the corresponding color image and depth map ;

[0056] From color image Extract local key point coordinates and , combined with the depth map Mapping the camera pose to 3D coordinates , forming a local three-dimensional feature item ;

[0057] Extract color image Global characteristics of , and the camera pose Binding, forming global feature items, is the pseudo-view camera rotation matrix;

[0058] Add local 3D feature items and global feature items to the feature database:

[0059] ;

[0060] ;

[0061] Where, 、 The updated 、 .

[0062] Constructing 2D-3D correspondences in combination with the target color map includes calculating the global eigenvector of the image to be processed and The cosine similarity score of :

[0063] ;

[0064] according to Sort in descending order and select the one with the highest score The local image feature set corresponding to the image to be processed is queried from the feature database, and the local feature matcher LightGlue is used to match the two-dimensional key points of the image to be processed with the local feature descriptors bound to the three-dimensional coordinates in the reference image to construct a 2D-3D correspondence relationship set. :

[0065] ;

[0066] Will As the initial interior point set :

[0067] .

[0068] Performing dynamic interior point screening and pose solving includes, in the In the round iteration, use Intra-wheel point set As input, the PnP-RANSAC algorithm is used to estimate the current camera pose matching pair , calculate the reprojection error for each matching pair :

[0069] ;

[0070] Where, Represents the camera projection function;

[0071] like Greater than the set threshold , remove the corresponding matching pairs from the internal point set and get the updated Intra-wheel point set ;

[0072] get Then, based on the matching pairs , use the trained main model to render the color image and depth image under the corresponding perspective, extract local features and re-match them with the local features of the image to be processed, and construct Round Enhanced Interior Point Set ;

[0073] Will With the current The point sets within the round are merged to obtain Intra-wheel point set:

[0074] ;

[0075] Where, It is The set of interior points of the round iteration;

[0076] The iteration process terminates when any of the following conditions is met:

[0077] (1) The intersection and union ratio of two consecutive rounds of inlier sets exceeds the set threshold :

[0078] ;

[0079] (2) The number of iterations exceeds the maximum setting value ;

[0080] Finally output the current optimal camera pose As the positioning result of the image to be processed.

[0081] Compared with the existing technology, the present invention has the following beneficial effects: the present invention intelligently fills in the missing perspectives of key areas, effectively improves the balance and coverage of feature expression in three-dimensional scenes, realizes the dynamic update of 2D-3D matching relationships and elimination of false matches, thereby significantly improving positioning accuracy and system stability, and has stronger robustness and practical usability under sparse training conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is the overall flow chart of the present invention;

[0083] Figure 2 Constructing a flow chart for the feature database of the present invention;

[0084] Figure 3 This is a flow chart of the visual enhancement method based on sparse perception;

[0085] Figure 4 Flowchart for dynamic interior point screening and pose solution. DETAILED DESCRIPTION

[0086] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention are described clearly and completely below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0087] A 3D Gaussian visual localization method for sparse viewpoints, including training a 3D Gaussian sputtering model using RGB images and camera poses, generating color images and depth maps through 2D projection and differentiable rendering mechanisms, extracting global and local image features from the sparse image, and building a feature database.

[0088] The feature database and RGB image are searched for the nearest neighbor to obtain the sparsity index, and the camera pose of the pseudo-perspective is calculated. The trained main model is used to render the color image and depth image under the pseudo-perspective. Global and local features are extracted and incorporated into the feature database. The 2D-3D correspondence relationship is constructed in combination with the target color image. Dynamic inlier screening and pose solution are performed to obtain the positioning result.

[0089] Training the three-dimensional Gaussian sputtering model includes: the RGB image is a sparse perspective image, the same sparse perspective image is used as the input of the three-dimensional Gaussian sputtering model, and the three-dimensional Gaussian sputtering model is constructed by a dual-branch three-dimensional Gaussian point cloud model collaborative modeling method, including training the main model and auxiliary supervision models , each three-dimensional Gaussian point cloud model includes multiple Gaussian points, and the Gaussian points are quadruples , 、 、 、 are the spatial position, the covariance matrix of the spatial scale and orientation of the Gaussian point, the color value and the opacity;

[0090] The same sparse view images are fed into the training main model and auxiliary supervision models , constructing cross-model view based on the rendered image under the same pseudo-perspective Figure 1 Consistency loss, loss function for:

[0091] ;

[0092] ;

[0093] Where, is the balance coefficient, and They are and Rendered image at the same pseudo perspective, represents the L1 norm, represents the structural similarity index;

[0094] Color reconstruction loss for a single 3D Gaussian point cloud model for:

[0095] ;

[0096] Where, is the pixel color of the training image, is the corresponding pixel value of the rendered image, Pixels , is the pixel color;

[0097] Total loss function for dual-branch training for:

[0098] ;

[0099] Where, is the weight coefficient, which is used to adjust the cross-model view Figure 1 The proportion of fatal losses in total losses;

[0100] The training of the 3D Gaussian sputtering model is completed by minimizing the total loss of the two-branch training.

[0101] Generating color images and depth maps involves projecting the three-dimensional covariance matrix into a two-dimensional covariance matrix based on the Jacobian transformation at the camera's perspective. :

[0102] ;

[0103] Where 𝐽 is the Jacobian matrix of the affine approximation of the projection function, is the view transformation matrix of the camera's viewing direction;

[0104] For each pixel in the image plane ,according to Sort the two-dimensional distances from all visible Gaussian points and construct a Gaussian set , using alpha-blending rendering to accumulate color values, generating ,Will As a color image:

[0105] ;

[0106] ;

[0107] Where, Gaussian point The cumulative transmittance, Gaussian point In pixels The shadow opacity, Indicates the Gaussian points, Indicates the Gaussian points, Gaussian point In pixels The shadow opacity;

[0108] Generate a depth map, the depth of a single pixel for:

[0109] ;

[0110] Where, Represents Gaussian points The absolute depth value at the current viewing angle will be As a depth map.

[0111] Constructing a feature database includes constructing a global image feature set and a local image feature set;

[0112] The global image feature set for:

[0113] ;

[0114] Where, is the first global eigenvectors, is the first A rotation matrix, is the first translation vector, yes The maximum value of

[0115] The local image feature set for:

[0116] ;

[0117] Where, is the three-dimensional space coordinate obtained by back-projection of local key points of the image, For Bound local feature descriptors.

[0118] Constructing a global image feature set includes extracting two-dimensional key point coordinates from sparse view images using an image local feature extraction network and corresponding , combined with the camera intrinsic parameter matrix corresponding to the sparse view image , external parameter matrix , the depth value of each key point in the depth map , map the two-dimensional key points to three-dimensional space and calculate the corresponding :

[0119] ;

[0120] Where, is the camera rotation matrix in the extrinsic matrix, is the translation vector in the external parameter matrix, Corresponding Binding to form local three-dimensional feature items And stored in the local image feature set.

[0121] Obtaining the sparsity index includes: Gaussian points in the three-dimensional Gaussian sputtering model ,statistics The visible perspective set , Equivalent to all observable Sparse view images, for the Sparse view images ,set up The set of observed Gaussian points is , The sparsity index for:

[0122] ;

[0123] Two images and The visual overlap of the two visible Gaussian point sets is the intersection ratio of the two visible Gaussian point sets. :

[0124] ;

[0125] Where, It is A sparse perspective image, yes The set of observed Gaussian points;

[0126] Using sparsity metrics Build a sparsity-sorted list.

[0127] Calculating the pseudo-perspective camera pose involves selecting the lowest sparsity from the sparsity sorted list As the target perspective to be enhanced, the one with the same of Greater than the set threshold Image collection :

[0128] ;

[0129] For The reference image with the farthest Euclidean distance in space, The posture is , is the translation vector, The rotation angle represented by the unit quaternion;

[0130] set up The camera pose is , the pseudo-view pose generated by interpolation is defined as:

[0131] ;

[0132] ;

[0133] ;

[0134] ;

[0135] Where, is the pseudo-perspective translation vector, is the pseudo-view rotation angle, is the interpolation factor, is the intermediate parameter.

[0136] Rendering color images and depth maps in pseudo-perspective using the trained main model includes using the same method as generating color images and depth maps, using the trained main model Camera pose in pseudo-perspective Generate the corresponding color image and depth map ;

[0137] From color image Extract local key point coordinates and , combined with the depth map Mapping the camera pose to 3D coordinates , forming a local three-dimensional feature item ;

[0138] Extract color image Global characteristics of , and the camera pose Binding, forming global feature items, is the pseudo-view camera rotation matrix;

[0139] Add local 3D feature items and global feature items to the feature database:

[0140] ;

[0141] ;

[0142] Where, 、 The updated 、 .

[0143] Constructing 2D-3D correspondences in combination with the target color map includes calculating the global eigenvector of the image to be processed and The cosine similarity score of :

[0144] ;

[0145] according to Sort in descending order and select the one with the highest score The local image feature set corresponding to the image to be processed is queried from the feature database, and the local feature matcher LightGlue is used to match the two-dimensional key points of the image to be processed with the local feature descriptors bound to the three-dimensional coordinates in the reference image to construct a 2D-3D correspondence relationship set. :

[0146] ;

[0147] Will As the initial interior point set :

[0148] .

[0149] Performing dynamic interior point screening and pose solving includes, in the In the round iteration, use Intra-wheel point set As input, the PnP-RANSAC algorithm is used to estimate the current camera pose matching pair , calculate the reprojection error for each matching pair :

[0150] ;

[0151] Where, represents the camera projection function;

[0152] like Greater than the set threshold , remove the corresponding matching pairs from the internal point set and get the updated Intra-wheel point set ;

[0153] get Then, based on the matching pairs , use the trained main model to render the color image and depth image under the corresponding perspective, extract local features and re-match them with the local features of the image to be processed, and construct Round Enhanced Interior Point Set ;

[0154] Will With the current The point sets within the round are merged to obtain Intra-wheel point set:

[0155] ;

[0156] Where, It is The set of interior points of the round iteration;

[0157] The iteration process terminates when any of the following conditions is met:

[0158] (1) The intersection and union ratio of two consecutive rounds of inlier sets exceeds the set threshold :

[0159] ;

[0160] (2) The number of iterations exceeds the maximum setting value ;

[0161] Finally output the current optimal camera pose As the positioning result of the image to be processed.

[0162] The overall process of the present invention is as follows Figure 1 As shown, it includes using RGB images and camera poses to train a three-dimensional Gaussian sputtering model, performing sparse perspective enhancement, extracting global image features and local image features of the sparse image, and building a feature database; performing nearest neighbor search on the feature database and the image to be processed to obtain a sparsity index, calculating the camera pose of the pseudo perspective, using the trained main model to render the color image and depth image under the pseudo perspective, extracting global features and local features and incorporating them into the feature database, building a 2D-3D correspondence relationship in combination with the target color image, performing dynamic interior point screening and pose solution, and obtaining positioning results. The feature database construction process of the present invention is as follows: Figure 2As shown in the figure, first obtain the RGB image and camera parameters, perform 3D Gaussian point cloud modeling, generate color image and depth image after rendering, use NetVLAD network to extract local features for color image, use SuperPoint network to extract local features for depth image, and finally obtain the constructed feature database. The process of visual enhancement method based on sparse perception is as follows: Figure 3 As shown in the figure, input the candidate perspective set, calculate the visibility score of each perspective, select the sparsest perspective, select the reference perspective, interpolate to generate pseudo perspective, extract the pseudo perspective features and add them to the feature database, and judge whether the end condition is met. If it is met, the calculation ends, if not, return to select the sparsest perspective. The pseudo perspective generated by interpolation needs to be returned to the candidate perspective set. The dynamic interior point screening and pose solution process is as follows: Figure 4 As shown, it includes inputting the image to be processed (query image) and the feature database, image retrieval to obtain the initial pose, extracting local features of the query image, performing 2D-3D matching, updating the 2D-3D correspondence, and solving the camera pose. If it does not converge, it returns to 2D-3D matching. If it converges, it outputs the camera pose.

[0163] In an embodiment of the present invention, the following steps are included:

[0164] S1: Obtain sparse view images and their camera pose information, build two 3D Gaussian point cloud models, train the main model and auxiliary supervision model respectively, and construct a cross-model view by minimizing the difference in the rendering results of the two models under the same view. Figure 1 The consistency loss is reduced to optimize the 3D structure modeling capability of the main model.

[0165] S2: Extract global feature vectors and local feature descriptors from sparse images. Use the depth information, camera intrinsic parameter matrix, and pose information corresponding to the image to map local feature points to three-dimensional space through the camera back-projection model. Build a feature database containing global features, three-dimensional coordinates of key points, and local features, and establish an inverted index structure to accelerate image retrieval.

[0166] S3: Based on the sparse perception perspective enhancement strategy proposed in this invention, the visibility distribution of each point in the three-dimensional Gaussian model under the training image perspective is statistically analyzed. The distribution density of the camera pose in the three-dimensional space and the overlap between the perspectives are combined to calculate the sparsity index of each training perspective. Based on this, a sparsity sorted list is generated for subsequent enhanced modeling of sparse areas.

[0167] S4: Select the top-ranked training view from the sparsity-ranked list. Based on other training images that partially overlap with the view coverage area and have significantly different spatial positions, calculate the camera pose of the pseudo-view by linear interpolation of the camera position and quaternion interpolation of the pose. The trained main model is then used to render the color and depth images under this pseudo-view. Global and local features are extracted and incorporated into the feature database to enhance the feature representation of sparse areas.

[0168] S5: Extract global features and local features from the query image, calculate the similarity between the global features of the query image and the set of global features in the database, retrieve several reference images with the highest similarity as candidate perspectives, and then perform descriptor matching on the local features of the query image and the local features bound to the three-dimensional coordinates in the candidate images to establish the initial correspondence between the two-dimensional key points in the query image and the three-dimensional space points.

[0169] S6: Based on the dynamic inlier selection and iterative optimization strategy proposed in this patent, the initial 2D-3D matching relationship is optimized using the PnP-RANSAC algorithm, using the reprojection error and 3D Gaussian rendering consistency metrics. In each iteration, incorrect matching points are dynamically removed based on the aforementioned metrics, the inlier set is updated, and the pose estimate is optimized until the inlier set converges, ultimately outputting the precise camera pose of the query image.

[0170] The present invention first obtains sparse training images and their corresponding camera pose information, constructs a Gaussian point cloud model based on three-dimensional Gaussian rendering technology, extracts global and local visual features of the image, and constructs a feature database. Subsequently, by analyzing the visibility distribution of Gaussian points, the areas with insufficient view coverage in the training set are identified, and pseudo-view camera poses are generated by combining reference view angles with distant spatial distributions but similar coverage areas. The Gaussian model is used to render the pseudo image and complete the feature database. In the positioning stage, after the query image is input, its global features are extracted, and image retrieval is performed to obtain candidate reference view angles. The initial 2D-3D correspondence is then constructed through local feature matching. Finally, a dynamic inlier screening and PnP-RANSAC optimization strategy guided by reprojection error and rendering consistency is adopted to iteratively estimate and output the final camera pose.

[0171] In constructing the feature database, the training images and their corresponding camera intrinsic and extrinsic parameter information are first obtained. Based on this image set and three-dimensional Gaussian modeling technology, a Gaussian point cloud representation of the scene is constructed. Each Gaussian point has attributes such as position, covariance matrix, color value, and opacity, which can be used in the subsequent differentiable rendering process. At the image level, the global visual features and local key point information of the training images are extracted separately. The global features are extracted using the NetVLAD network for subsequent image retrieval, and the local features are extracted using the SuperPoint network to extract two-dimensional key points and their descriptors. Combining the camera pose and the depth map obtained by rendering, the key points are back-projected into three-dimensional space to generate a set of local three-dimensional features that bind three-dimensional coordinates to local descriptors. After construction is completed, the above global and local features are organized into feature databases, where local features are used for two-dimensional-to-three-dimensional matching and global features are used for image retrieval.

[0172] The sparse perception enhancement strategy aims to address the problem of missing geometric information in local areas caused by the uneven distribution of training image perspectives. This method evaluates the perspective sparsity of each training image based on the visibility distribution information of the point cloud in a three-dimensional Gaussian model. Specifically, for each three-dimensional Gaussian point, the set of training images in which it is observed is counted. The average visibility score is then calculated for each training image to form a sparsity-ranked list. The perspective with the highest sparsity is selected as the enhancement target. To expand the spatial information coverage of this sparse perspective, the method further selects training images that are relatively far apart in three-dimensional space but have a high degree of overlap in the observed area as reference perspectives, and uses interpolation to generate a pseudo-perspective camera pose. Under the generated pseudo-perspective, the trained three-dimensional Gaussian point cloud model is used to render the color image and depth map, and the corresponding global and local visual features are extracted. Finally, the feature data items of this pseudo-perspective are added to the feature database. This enhancement process can be iterated until the visibility scores of all perspectives reach a set threshold or the number of generated pseudo-perspectives reaches a set upper limit, thereby achieving structural completion and expression enhancement of the original sparse perspective space.

[0173] The dynamic inlier screening and pose solution process primarily establishes a stable correspondence between 2D keypoints and 3D spatial points after a query image input, and improves the accuracy and robustness of camera pose estimation through dynamic optimization. First, global features are extracted from the query image to be located and similarity matched against the global features of training images stored in the feature database. Several images with the highest scores are retrieved as candidate reference viewpoints. Furthermore, local keypoints and descriptors of the query image are extracted and locally matched against the bound 3D local features of the candidate reference images to construct an initial set of 2D-3D matches. Subsequently, the present invention introduces a dynamic inlier screening mechanism, employing an iterative PnP-RANSAC process to estimate the camera pose. In each iteration, the current inlier set is updated by combining reprojection error and rendering consistency error. After each round of optimization, the estimated camera pose is used to render the view on a 3D Gaussian model, and local features from the new viewpoint are extracted to guide the generation of new 2D-3D matching points, thereby replenishing potential missed matches. The above process terminates when the inner point set convergence condition is met or the maximum number of iterations is reached, and the optimized camera pose result is finally output. This module effectively alleviates the matching failure problem caused by initial pose error or occlusion interference, significantly improving the robustness and accuracy of positioning under sparse view conditions.

[0174] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D Gaussian vision positioning method for sparse view, characterized in that: This includes training a 3D Gaussian sputtering model using RGB images and camera poses, generating color images and depth maps through 2D projection and differentiable rendering mechanisms, extracting global and local image features from sparse images, and building a feature database. The feature database and RGB image are searched for the nearest neighbor to obtain the sparsity index, the camera pose under the pseudo-perspective is calculated, the trained main model is used to render the color image and depth image under the pseudo-perspective, the global and local features are extracted and incorporated into the feature database, and the 2D-3D correspondence is constructed in combination with the target color image. Dynamic inlier screening and pose solution are performed to obtain the positioning result. Get the sparsity index Including, for Gaussian points in the three-dimensional Gaussian sputtering model ,statistics The visible perspective set , Equivalent to all observable For the sparse view image of Sparse view images ,set up The set of observed Gaussian points is , The sparsity index for: ； Two images and The visual overlap of the two visible Gaussian point sets is the intersection ratio of the two visible Gaussian point sets. : ； Where, It is A sparse perspective image, yes The set of observed Gaussian points; Using sparsity metrics Construct a sparsity-sorted list; Calculating the pseudo-perspective camera pose involves selecting the lowest sparsity from the sparsity sorted list As the target perspective to be enhanced, the one with the same of Greater than the set threshold Image collection : ； For The reference image with the farthest Euclidean distance in space, The posture is , is the translation vector, The rotation angle represented by the unit quaternion; set up The camera pose is , the pseudo-view pose generated by interpolation is defined as: ； ； ； ； Where, is the pseudo-perspective translation vector, is the pseudo-view rotation angle, is the interpolation factor, is the intermediate parameter; Performing dynamic interior point screening and pose solving includes, in the In the round iteration, use Intra-wheel point set As input, the PnP-RANSAC algorithm is used to estimate the current camera pose matching pair , calculate the reprojection error for each matching pair : ； Where, represents the camera projection function, is the camera intrinsic parameter matrix, are the two-dimensional keypoint coordinates, is the three-dimensional space coordinate obtained by back-projection of local key points of the image; like Greater than the set threshold , remove the corresponding matching pairs from the internal point set and get the updated Intra-wheel point set ; get Then, based on the matching pairs , use the trained main model to render the color image and depth image under the corresponding perspective, extract local features and re-match them with the local features of the image to be processed, and construct Round Enhanced Interior Point Set , is the camera rotation matrix in the extrinsic matrix, is the translation vector in the extrinsic matrix; Will With the current The point sets within the round are merged to obtain Intra-wheel point set: ； Where, It is The set of interior points of the round iteration; The iteration process terminates when any of the following conditions is met: (1) The intersection and union ratio of two consecutive rounds of inlier sets exceeds the set threshold : ； (2) The number of iterations exceeds the maximum setting value ; Finally output the current optimal camera pose As the positioning result of the image to be processed.

2. The sparse-view oriented three-dimensional Gaussian vision positioning method according to claim 1, characterized in that: Training the three-dimensional Gaussian sputtering model includes: the RGB image is a sparse perspective image, the same sparse perspective image is used as the input of the three-dimensional Gaussian sputtering model, and the three-dimensional Gaussian sputtering model is constructed by a dual-branch three-dimensional Gaussian point cloud model collaborative modeling method, including training the main model and auxiliary supervision models , each three-dimensional Gaussian point cloud model includes multiple Gaussian points, and the Gaussian points are quadruples , 、 、 、 are the spatial position, the covariance matrix of the spatial scale and orientation of the Gaussian point, the color value and the opacity; The same sparse view images are fed into the training main model and auxiliary supervision models , based on the rendered image under the same pseudo-perspective, the cross-model view consistency loss is constructed, and the loss function for: ； ； Where, is the balance coefficient, and They are and Rendered image at the same pseudo perspective, represents the L1 norm, represents the structural similarity index; Color reconstruction loss for a single 3D Gaussian point cloud model for: ； Where, is the pixel color of the training image, is the corresponding pixel value of the rendered image, Pixels , is the pixel color; Total loss function for dual-branch training for: ； Where, is the weight coefficient, which is used to adjust the proportion of cross-model view consistency loss in the total loss; The training of the 3D Gaussian sputtering model is completed by minimizing the total loss of the two-branch training.

3. The sparse-view oriented three-dimensional Gaussian vision positioning method according to claim 2, characterized in that: Generating color images and depth maps involves projecting the three-dimensional covariance matrix into a two-dimensional covariance matrix based on the Jacobian transformation at the camera's perspective. : ； Where 𝐽 is the Jacobian matrix of the affine approximation of the projection function, is the view transformation matrix of the camera's viewing direction; For each pixel in the image plane ,according to Sort the two-dimensional distances from all visible Gaussian points and construct a Gaussian set , using alpha-blending rendering to accumulate color values, generating ,Will As a color image: ； ； Where, Gaussian point The cumulative transmittance, Gaussian point In pixels The shadow opacity, Indicates the Gaussian points, Indicates the Gaussian points, Gaussian point In pixels The shadow opacity; Generate a depth map, the depth of a single pixel for: ； Where, Represents Gaussian points The absolute depth value at the current viewing angle will be As a depth map.

4. The sparse-view oriented three-dimensional Gaussian vision positioning method according to claim 3, characterized in that: Constructing a feature database includes constructing a global image feature set and a local image feature set; The global image feature set for: ； Where, is the first global eigenvectors, is the first A rotation matrix, is the first translation vector, yes The maximum value of The local image feature set for: ； Where, For Bound local feature descriptors.

5. The sparse-view oriented three-dimensional Gaussian vision positioning method according to claim 4, characterized in that: Constructing a global image feature set includes extracting two-dimensional key point coordinates from sparse view images using an image local feature extraction network and corresponding , combined with the camera intrinsic parameter matrix corresponding to the sparse view image , external parameter matrix , the depth value of each key point in the depth map , map the two-dimensional key points to three-dimensional space and calculate the corresponding : ； In the formula, Corresponding Binding to form local three-dimensional feature items And stored in the local image feature set.

6. The sparse-view oriented three-dimensional Gaussian vision positioning method according to claim 5, characterized in that: Rendering color images and depth maps in pseudo-perspective using the trained main model includes using the same method as generating color images and depth maps, using the trained main model Camera pose in pseudo-perspective Generate the corresponding color image and depth map ; From color image Extract local key point coordinates and , combined with the depth map Mapping the camera pose to 3D coordinates , forming a local three-dimensional feature item ; Extract color image Global characteristics of , and the camera pose Binding, forming global feature items, is the pseudo-view camera rotation matrix; Add local 3D feature items and global feature items to the feature database: ； ； Where, 、 The updated 、 .

7. The sparse-view oriented three-dimensional Gaussian vision positioning method according to claim 6, characterized in that: Constructing 2D-3D correspondences in combination with the target color map includes calculating the global eigenvector of the image to be processed and The cosine similarity score of : ； according to Sort in descending order and select the one with the highest score The local image feature set corresponding to the image to be processed is queried from the feature database, and the local feature matcher LightGlue is used to match the two-dimensional key points of the image to be processed with the local feature descriptors bound to the three-dimensional coordinates in the reference image to construct a 2D-3D correspondence relationship set. : ； Will As the initial interior point set : 。

Images