A deep learning-based visual global position recognition and positioning method

Abstract

The application discloses a kind of visual global position recognition and positioning method based on deep learning, by introducing high discriminative global feature extraction network, image scene is efficiently coded, so that each frame image can be accurately and compactly expressed, to significantly improve the reliability and accuracy of position matching.Meanwhile, the application uses efficient feature matching strategy, and the scene feature database is quickly compared with the image to be positioned, and the candidate matching frame is quickly screened, so that the calculation efficiency of the positioning process is greatly improved.Based on the candidate frame, the application further introduces geometric consistency verification and local feature fine matching, effectively eliminates the error matching, and guarantees the stability and accuracy of pose estimation.In addition, the application optimizes the matching result by combining the global pose optimization of nonlinear least square method and the incremental optimization of sliding window, effectively reduces the cumulative error, and improves the continuity and reliability of long sequence positioning.

Description

Technical Field

[0001] This invention relates to the field of robot localization technology, specifically to a deep learning-based visual global position recognition and localization method. Background Technology

[0002] With the rapid development of intelligent robots, autonomous vehicles, drones, and mobile terminals, positioning and navigation technologies have become crucial foundations for achieving autonomous mobility and environmental awareness. Among various positioning schemes, visual positioning has attracted widespread attention due to its ability to acquire rich environmental information using conventional cameras and its low hardware cost. Existing visual positioning methods mainly include those based on local feature matching and those based on end-to-end deep learning regression.

[0003] Local feature matching-based methods extract keypoint features from images, such as SIFT, ORB, or SURF descriptors, and match these features with features in the mapped scene to calculate the camera's pose in 3D space. While this method can achieve high positioning accuracy in small-scale environments, it has significant limitations in practical applications. Local feature matching is prone to failure when lighting conditions change, dynamic occlusions exist, or the camera's shooting angle differs significantly from the mapping stage, leading to decreased positioning accuracy. Furthermore, the computational demands of feature detection, descriptor calculation, and matching are substantial, making real-time operation on embedded devices and mobile platforms unsuitable.

[0004] End-to-end deep learning regression methods directly map input images to camera pose vectors by constructing convolutional neural networks, deformable networks, or combinations thereof. These methods can alleviate the sensitivity of traditional methods to environmental changes to some extent and reduce reliance on manually designed features. However, existing regression methods also have significant drawbacks. The models lack generalization ability in scenes outside the training environment, easily leading to localization errors; they require large amounts of training data covering different lighting, weather, seasons, and viewpoints, resulting in high data collection costs; they are prone to cumulative errors in long-sequence localization tasks, making it difficult to guarantee long-term stability; and they lack mechanisms to fully utilize scene geometry, exhibiting low sensitivity to small-scale pose changes.

[0005] In practical applications, visual positioning systems need to balance high accuracy, robustness, and real-time performance, especially in scenarios such as autonomous driving, warehouse robots, drone navigation, and augmented reality. However, existing technologies still have many shortcomings: stable positioning is difficult in complex environments, high computational resource requirements are placed on embedded and mobile devices, and real-time performance is insufficient; there is a lack of a unified global feature representation and pose optimization framework between the mapping and localization stages, making the system susceptible to noise and accumulating errors; and there is a trade-off between efficiency and accuracy in feature matching and pose estimation.

[0006] Therefore, there is an urgent need for a new visual positioning method that can achieve a high-precision, robust, and real-time visual positioning solution in complex environments by combining highly discriminative global feature extraction and efficient matching with geometric consistency verification and pose optimization strategies. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems existing in the prior art by providing a deep learning-based visual global position recognition and localization method. This addresses the issues of low localization accuracy, poor robustness, and insufficient real-time performance in complex environments found in current visual localization methods. This invention introduces a highly discriminative global feature extraction network to efficiently encode image scenes, ensuring that each frame is accurately and compactly represented, thereby significantly improving the reliability and accuracy of position matching. Simultaneously, this invention employs an efficient feature matching strategy to quickly compare the image to be located with a scene feature database, rapidly filtering candidate matching frames and greatly improving the computational efficiency of the localization process. Based on candidate frames, this invention further introduces geometric consistency verification and fine-grained local feature matching to effectively eliminate erroneous matches, ensuring the stability and accuracy of pose estimation. Furthermore, this invention optimizes the matching results by combining global pose optimization using nonlinear least squares and incremental optimization using a sliding window, effectively reducing accumulated errors and improving the continuity and reliability of long-sequence localization.

[0008] To achieve the above objectives, this invention provides a deep learning-based visual global location recognition and localization method, comprising the following steps:

[0009] S1. Collect multi-view image sequences of the scene to be mapped, and preprocess the collected images. The preprocessing includes brightness normalization, color equalization, lens distortion correction, scale adjustment and noise reduction to improve the stability and robustness of subsequent feature extraction.

[0010] S2. Input the preprocessed image into the global feature extraction network. The global feature extraction network is a model built based on deep learning. Through multi-layer feature extraction and global aggregation operations, it obtains the global feature descriptor corresponding to each frame of the image.

[0011] S3. The global feature descriptor is reduced in dimensionality, normalized and compressed. Feature quantization encoding and hash mapping are used to reduce storage and retrieval computation. The processed descriptor and the corresponding image pose information are stored in the scene feature database.

[0012] S4. Obtain the image to be located and perform the same preprocessing operation as in step S1 to ensure that the feature extraction input is consistent.

[0013] S5. Input the image to be located into the global feature extraction network in step S2 to obtain the global feature descriptor of the image, and perform similarity calculation with the descriptor in the scene feature database. The similarity calculation adopts the cosine similarity calculation method, and selects several candidate matching frames based on the similarity score.

[0014] S6. Perform geometric consistency verification on candidate matching frames. The geometric consistency verification includes feature point-based matching, random consistency sampling model fitting, and reprojection error calculation. Eliminate erroneous matches that do not meet geometric constraints to obtain the final set of valid matching frames.

[0015] S7. Based on the set of valid matching frames, the camera pose of the image to be located is solved using a pose optimization algorithm. The pose optimization algorithm adopts global optimization based on nonlinear least squares method and outputs the three-dimensional position and pose information of the image.

[0016] Furthermore, step S1 specifically includes the following steps:

[0017] S11. Collect multi-view image sequences of the scene to be mapped and preprocess the collected images;

[0018] S12. Eliminate light intensity and color deviation through brightness normalization and color balance algorithms to ensure consistent image brightness and color under different shooting conditions;

[0019] S13. Correct the distortion of the image according to the camera calibration parameters and scale the image to a uniform resolution to improve the accuracy and computational efficiency of subsequent feature extraction.

[0020] Furthermore, step S2 specifically includes the following steps:

[0021] S21. Input the preprocessed image into the constructed deep neural network model, and extract multi-scale image feature representations through multi-layer convolution and feature encoding;

[0022] S22. Perform global pooling and attention aggregation on the extracted features to compress the spatial dimension into a single vector representation, thus obtaining the global feature descriptor of the image.

[0023] Furthermore, step S3 specifically includes the following steps:

[0024] S31. The high-dimensional descriptor is reduced in dimensionality by using feature quantization encoding method, and L2 normalization is used to ensure consistent feature vector distribution, thereby improving the stability of similarity calculation.

[0025] S32. Use vector quantization to compress the dimensionality-reduced features, reducing storage space and computational load for subsequent retrieval.

[0026] S33. Store the compressed feature vector and its corresponding image pose information together in the scene feature database for subsequent retrieval and localization calculations.

[0027] Furthermore, step S4 specifically includes the following steps:

[0028] S41. Obtain the image to be located and perform the same preprocessing operations as in step S1 (brightness normalization, distortion correction, and scale adjustment) on it to ensure that the input features are consistent with those in the training phase.

[0029] Furthermore, step S5 specifically includes the following steps:

[0030] S51. Input the image to be located into the global feature extraction network to obtain the global feature descriptor corresponding to the image;

[0031] S52. The extracted global feature descriptors are compared with the descriptors in the scene feature database. The similarity calculation method uses cosine similarity. During the calculation process, parallel calculation is performed through a graphics processor or a central processing unit to accelerate the calculation and reduce the matching time.

[0032] S53. Sort the images in the database according to the similarity score, and select several candidate matching frames with the highest scores for subsequent localization and matching.

[0033] Furthermore, step S6 specifically includes the following steps:

[0034] S61. Local key point features are extracted from the image to be located and each candidate matching frame through a feature model. Corresponding point pairs are established between the two images through descriptor matching methods (such as nearest neighbor matching or cross-validation matching) to provide a basis for geometric verification.

[0035] S62. Using the random consistency sampling algorithm, a geometric model is fitted to the feature point matching pairs between the candidate matching frame and the image to be located. During the iteration process, outlier matching points that do not conform to the geometric model are automatically removed to improve the robustness of the matching and determine the interior point set of the optimal geometric model, providing a reliable matching basis for subsequent reprojection error calculation.

[0036] S63. Using the fitted geometric model, project the key points of the image to be localized onto candidate frames and calculate the reprojection error. Set an error threshold and remove matching frames or matching point pairs whose reprojection errors exceed the threshold. The remaining matching frames constitute the final set of valid matching frames for subsequent accurate pose estimation.

[0037] Furthermore, step S7 specifically includes the following steps:

[0038] S71. Based on the correspondence of key points extracted from the set of effective matching frames, construct the pose optimization problem of the image to be located, take the camera pose parameters (rotation matrix and translation vector) as the variables to be optimized, and take the reprojection error of the matching points as the objective function to ensure that the optimization process can consider the geometric constraints of all effective matching frames at the same time.

[0039] S72. Use a global optimization algorithm based on nonlinear least squares (such as the Levenburg-Marquardt algorithm or the Gauss-Newton algorithm) to iteratively solve the optimal camera pose. By continuously adjusting the rotation and translation parameters, minimize the reprojection error of all matching points, thereby improving the accuracy and robustness of pose estimation.

[0040] S73. After optimization, the obtained camera rotation matrix and translation vector are converted into three-dimensional position and attitude information output to provide accurate camera pose results for subsequent localization, navigation or scene understanding tasks.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] 1. This invention introduces a deep learning-based global feature extraction network to encode scene images into highly discriminative feature descriptors, enabling each frame to be accurately represented in a low-dimensional space. These global features effectively distinguish similar scenes during feature matching, reducing the possibility of false matches. Simultaneously, by combining geometric consistency verification and fine-grained local feature matching steps, erroneous frames in candidate matches can be eliminated, significantly improving the accuracy of camera pose estimation. Experiments show that even in environments with drastic lighting changes or significant viewing angle differences, this method can maintain low positioning errors, achieving high-precision positioning from millimeter to centimeter level.

[0043] 2. The global feature extraction method proposed in this invention can capture the overall structural information of the scene, rather than simply relying on local feature points, thus maintaining strong recognition capabilities even in complex environments such as dynamic occlusion, seasonal changes, weather changes, and large viewpoint shifts. Furthermore, by introducing geometric verification and pose optimization, this invention can effectively eliminate abnormal matching and noise interference, ensuring that the system does not drift or misjudge during continuous localization, achieving long-term stable visual localization.

[0044] 3. This invention introduces efficient feature compression and fast retrieval strategies, such as feature quantization, dimensionality reduction, and approximate nearest neighbor search, into the global feature extraction, compression, and matching processes, significantly reducing computational and storage overhead. Simultaneously, the end-to-end process design ensures minimal latency from image acquisition to pose output, enabling the system to achieve millisecond-level response in resource-constrained hardware environments, meeting the stringent real-time requirements of applications such as robot navigation, drone flight, and mobile terminal augmented reality.

[0045] 4. In the process of fine matching of local features, this invention eliminates matching frames and points that do not conform to the geometric model through geometric consistency verification, thereby avoiding the impact of incorrect matching on the localization results. Especially in complex scenes, dynamic objects, lighting changes and viewpoint differences, these can cause large errors in the localization results. Geometric consistency verification can effectively filter these incorrect matches based on image information, ensuring that the final matching results have sufficient accuracy.

[0046] 5. In its implementation, this invention adopts a "coarse-to-fine" approach. First, it uses global features for matching to obtain candidate frames, and then performs high-precision positioning by comparing these candidate frames with the frame to be located. This method first uses global features for rapid matching, quickly filtering out the most relevant candidate frames, thus significantly reducing the computational load of subsequent fine-grained matching. Next, it further processes the candidate frames using precise matching of local features and geometric consistency verification to ensure high accuracy in the final matching result. This "coarse-to-fine" matching strategy not only improves positioning accuracy but also significantly reduces positioning time, increasing system efficiency. Through this phased processing approach, this invention can improve the real-time performance of the system while maintaining accuracy, especially in dynamically changing environments, enabling efficient completion of positioning tasks in complex scenarios and meeting the dual requirements of real-time performance and high accuracy for applications such as autonomous driving, drone flight, and robot navigation.

[0047] 6. This invention designs a complete closed-loop self-localization process, from mapping, feature extraction, database construction, candidate matching, geometric verification to pose optimization, forming an end-to-end visual localization solution. This process not only ensures localization accuracy and robustness but also provides high scalability, enabling it to be used in conjunction with different types of deep learning network architectures, feature matching strategies, and pose optimization algorithms, achieving flexible system deployment. Furthermore, the complete process design makes the system easily integrated into various practical application scenarios such as autonomous driving, drone navigation, warehouse robots, and augmented reality, providing reliable technical support for visual localization in complex environments. Attached Figure Description

[0048] Figure 1 A schematic diagram of a deep learning-based visual global position recognition and localization method provided by the present invention;

[0049] Figure 2 A diagram illustrating the structure of a deep neural network provided by this invention;

[0050] Figure 3 This is a schematic diagram of the top 1 candidate frame matching result obtained by global image matching provided by the present invention.

[0051] Figure 4The image shows the pixel matching result obtained after feature point extraction and matching, as provided by this invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Figure 1 This is a schematic diagram of a deep learning-based visual global location recognition and localization method according to the present invention. Figure 1 As shown, the implementation steps of the deep learning-based visual global location recognition and localization method proposed in this invention are as follows:

[0054] Step S1: Collect multi-view image sequences of the scene to be mapped, and preprocess the collected images. The preprocessing includes brightness normalization, color equalization, lens distortion correction, scale adjustment and noise reduction to improve the stability and robustness of subsequent feature extraction.

[0055] Step S2: Input the preprocessed image into the global feature extraction network. The global feature extraction network is a model built based on deep learning. Through multi-layer feature extraction and global aggregation operations, it obtains the global feature descriptor corresponding to each frame of the image.

[0056] Step S3: The global feature descriptor is reduced in dimensionality, normalized and compressed. Feature quantization encoding and hash mapping are used to reduce storage and retrieval computation. The processed descriptor and the corresponding image pose information are stored in the scene feature database.

[0057] Step S4: Obtain the image to be located and perform the same preprocessing operation as in step S1 to ensure that the feature extraction input is consistent;

[0058] Step S5: Input the image to be located into the global feature extraction network in step S2 to obtain the global feature descriptor of the image, and perform similarity calculation with the descriptor in the scene feature database. The similarity calculation adopts the cosine similarity calculation method, and selects several candidate matching frames based on the similarity score.

[0059] Step S6: Perform geometric consistency verification on the candidate matching frames. The geometric consistency verification includes feature point-based matching, random consistency sampling model fitting, and reprojection error calculation. False matches that do not meet the geometric constraints are eliminated to obtain the final set of valid matching frames.

[0060] Step S7: Based on the set of valid matching frames, the camera pose of the image to be located is solved using a pose optimization algorithm. The pose optimization algorithm adopts global optimization based on nonlinear least squares method and outputs the three-dimensional position and pose information of the image.

[0061] Furthermore, the method for visual global location recognition and localization based on deep learning proposed in this invention includes the following specific steps in step S1:

[0062] Step S1.1: Collect multi-view image sequences of the scene to be mapped, and preprocess the collected images;

[0063] First, the camera will collect image data of the scene to be mapped and located. The acquisition frequency can be set to 10Hz, and the interval between each image can be controlled within 1m. After the acquisition is completed, the images will be stored in the device.

[0064] Step S1.2: Eliminate light intensity and color deviation through brightness normalization and color balance algorithms to ensure consistent image brightness and color under different shooting conditions.

[0065] Step S1.3: Correct the distortion of the image according to the camera calibration parameters and scale the image to a uniform resolution to improve the accuracy and computational efficiency of subsequent feature extraction.

[0066] For image distortion correction, the pixel coordinates (u,v) of the ideal distortion-free image are converted into normalized coordinates (x,y) in the camera coordinate system:

[0067]

[0068] in , Principal point coordinates , This is the camera's focal length. This step maps "image plane pixels" to "points on the plane perpendicular to the camera's optical axis".

[0069] Furthermore, the visual global position recognition and localization method based on deep learning proposed in this invention includes the following specific steps in step S2:

[0070] Step S2.1: Input the preprocessed image into the constructed deep neural network model, and extract multi-scale image feature representations through multi-layer convolution and feature encoding;

[0071] For the processed image, this invention uses the currently robust feature extraction model DINOv2 to extract image features, thereby obtaining a two-dimensional image feature representation.

[0072] The image feature extraction model code is shown below:

[0073] ```python

[0074] import torch

[0075] import torch.nn as nn

[0076] class DinoV2(nn.Module):

[0077] AVAILABLE_MODELS = [

[0078] 'dinov2_vits14',

[0079] 'dinov2_vitb14',

[0080] 'dinov2_vitl14',

[0081] 'dinov2_vitg14' ]

[0083] def __init__(

[0084] self,

[0085] backbone_name="dinov2_vits14",

[0086] num_unfrozen_blocks=2,

[0087] return_cls_token=False, ):

[0089] """DinoV2 backbone with the ability to keep only the lastnum_unfrozen_blocks trainable.

[0090] Args:

[0091] backbone_name (str, optional): DinoV2 variant. Defaultsto "dinov2_vitb14".

[0092] num_unfrozen_blocks (int, optional): number of blocks tounfreeze. Defaults to 2.

[0093] Raises:

[0094] ValueError: if the backbone_name is not in the availablemodels.

[0095] """

[0096] super().__init__()

[0097] self.backbone_name = backbone_name

[0098] self.num_unfrozen_blocks = num_unfrozen_blocks

[0099] self.return_cls_token = return_cls_token

[0100] # make sure the backbone_name is in the available models

[0101] if self.backbone_name not in self.AVAILABLE_MODELS:

[0102] raise ValueError(f"Backbone {self.backbone_name} is notrecognized!"

[0103] f"Supported backbones are:{self.AVAILABLE_MODELS}")

[0104] self.dino = torch.hub.load('facebookresearch / dinov2',self.backbone_name)

[0105] # freeze the patch embedding and positional encoding

[0106] self.dino.patch_embed.requires_grad_(False)

[0107] self.dino.pos_embed.requires_grad_(False)

[0108] # freeze the first blocks, keep only the last num_unfrozen_blocks trainable

[0109] for i in range(len(self.dino.blocks) - self.num_unfrozen_blocks):

[0110] self.dino.blocks[i].requires_grad_(False)

[0111] self.out_channels = self.dino.embed_dim

[0112] def forward(self, x):

[0113] B, _, H, W = x.shape

[0114] # No need to compute gradients for frozen layers

[0115] with torch.no_grad():

[0116] x = self.dino.prepare_tokens_with_masks(x)

[0117] for blk in self.dino.blocks[ : -self.num_unfrozen_blocks]:

[0118] x = blk(x)

[0119] x = x.detach()

[0120] # Last blocks are trained

[0121] for blk in self.dino.blocks[-self.num_unfrozen_blocks : ]:

[0122] x = blk(x)

[0123] x_cls = x[:, 0]

[0124] x = x[:, 1:] # remove the [CLS] token

[0125] # reshape the output tensor to B, C, H, W

[0126] _, _, C = x.shape # we know C == self.dino.embed_dim, butstill...

[0127] x = x.permute(0, 2, 1).contiguous().view(B, C, H / / 14, W / / 14)

[0128] if self.return_cls_token:

[0129] return x, x_cls

[0130] return x

[0131] ```

[0132] The specific process involves first dividing the image into 14... The image is divided into 14 pixel blocks, and then the relationship between pixels and within each block is calculated by a neural network to obtain the features of each block.

[0133] Step S2.2 involves performing global pooling and attention aggregation on the extracted features to compress the spatial dimension into a single vector representation, thereby obtaining the global feature descriptor of the image.

[0134] After obtaining the two-dimensional features of the image, a feature aggregation model is used to compress the two-dimensional features into a one-dimensional vector. Since some two-dimensional features are lost during the compression process, this invention uses global pooling and attention mechanisms to extract some important features. First, a global pooling layer is used to extract the confidence of the features. Static objects and scenes have higher confidence, while dynamic objects have lower confidence. Then, the confidence and the original features are fused through an attention mechanism to obtain the final feature vector description of the image.

[0135] The code for the image feature aggregation model is shown below:

[0136] ```

[0137] import math

[0138] import torch

[0139] import torch.nn as nn

[0140] import torch.nn.functional as F

[0141] def flatten_hw(x):

[0142] # x: B,C,H,W -> B, N(=H W), C

[0143] B, C, H, W = x.shape

[0144] return x.flatten(we need to correct this to 2).permute(we need to correct this to 0, 2, 1), H, W

[0145] # -----------------------------

[0146] # Positional encoding: Sinusoidal

[0147] # ----------------------------- [[ID=​​​​​​​​​​​​​​​​​​​​​​​​pe = torch.zeros((d_model, max_shape))

[0157] y_position = torch.arange(max_shape[0]).float().unsqueeze(1).unsqueeze(0) + 1

[0158] x_position = torch.arange(max_shape[1]).float().unsqueeze(0).unsqueeze(0) + 1

[0159] if temp_bug_fix:

[0160] div_term = torch.exp(torch.arange(0, d_model / / 2, 2).float() (-math.log(10000.0) / (d_model / / 2)))

[0161] else:

[0162] div_term = torch.exp(torch.arange(0, d_model / / 2, 2).float() (-math.log(10000.0) / d_model / / 2))

[0163] div_term = div_term[:, None, None] # [C / / 4,1,1]

[0164] pe[0::4, :, :] = torch.sin(x_position div_term)

[0165] pe[1::4, :, :] = torch.cos(x_position div_term)

[0166] pe[2::4, :, :] = torch.sin(y_position div_term)

[0167] pe[3::4, :, :] = torch.cos(y_position div_term)

[0168] self.register_buffer('pe', pe.unsqueeze(0), persistent=False)# [1, C, H, W]

[0169] def forward(self, x):

[0170] B, C, H, W = x.shape

[0171] # If the feature map is different from max_shape, perform interpolation

[0172] if H != self.pe.size(2) or W != self.pe.size(3):

[0173] pe = F.interpolate(self.pe, size=(H, W), mode='bilinear',align_corners=False)

[0174] else:

[0175] pe = self.pe

[0176] return x + pe

[0177] # -----------------------------

[0178] # Basic BoQBlock

[0179] # -----------------------------

[0180] class BoQBlock(nn.Module):

[0181] def __init__(self, dim, num_queries, nheads=8, dim_ff=4):

[0182] super().__init__()

[0183] self.encoder = nn.TransformerEncoderLayer(d_model=dim, nhead=nheads, dim_feedforward=dim_ff dim, batch_first=True, dropout=0.)

[0184] self.num_queries = num_queries

[0185] self.self_attn = nn.MultiheadAttention(dim, num_heads=nheads,batch_first=True)

[0186] self.norm_q = nn.LayerNorm(dim)

[0187] self.cross_attn = nn.MultiheadAttention(dim, num_heads=nheads, batch_first=True)

[0188] self.norm_out = nn.LayerNorm(dim)

[0189] def forward(self, x, q):

[0190] # x: B, N, C ; q: B, Q, C

[0191] x = self.encoder(x)

[0192] q = q + self.self_attn(q, q, q)[0]

[0193] q = self.norm_q(q)

[0194] out, attn = self.cross_attn(q, x, x)

[0195] out = self.norm_out(out)

[0196] return x, out, attn.detach()

[0197] # -----------------------------

[0198] # DynamicBoQ

[0199] # -----------------------------

[0200] class DynamicBoQ(nn.Module):

[0201] def __init__(self, in_channels=1024, proj_channels=512, num_queries=32, num_layers=2, row_dim=32, query_gen_hidden=512):

[0202] super().__init__()

[0203] # Projection convolution

[0204] self.proj = nn.Conv2d(in_channels, proj_channels, kernel_size=3, padding=1)

[0205] self.norm_input = nn.LayerNorm(proj_channels)

[0206] self.num_queries = num_queries

[0207] self.proj_channels = proj_channels

[0208] # Query generator

[0209] self.gempool = GeM()

[0210] self.query_gen = nn.Sequential(

[0211] nn.Linear(proj_channels, query_gen_hidden),

[0212] nn.GELU(),

[0213] nn.Linear(query_gen_hidden, num_queries proj_channels) )

[0215] # Positional Encoding

[0216] self.pos_encoding = PositionEncodingSine(proj_channels)

[0217] # BoQ blocks

[0218] nheads = max(1, proj_channels / / 64)

[0219] self.boqs = nn.ModuleList([BoQBlock(proj_channels, num_queries, nheads=nheads) for _ in range(num_layers)])

[0220] # Output Head

[0221] self.fc = nn.Sequential(

[0222] nn.LayerNorm(num_layers num_queries),

[0223] nn.Linear(num_layers num_queries, 4 row_dim),

[0224] nn.GELU(),

[0225] nn.Linear(4 row_dim, row_dim) )

[0227] def forward(self, feats):

[0228] # feats: B,C,H,W

[0229] x = self.proj(feats) # B,C,H,W

[0230] x = self.pos_encoding(x) # B,C,H,W

[0231] # flatten + permute -> [B, N, C]

[0232] B, C, H, W = x.shape

[0233] x_tokens = x.flatten(2).permute(0, 2, 1) # B, N=H W, C

[0234] x_tokens = self.norm_input(x_tokens)

[0235] # generate queries

[0236] pooled = self.gempool(x).flatten(1) # B, C

[0237] q = self.query_gen(pooled).view(B, self.num_queries,self.proj_channels)

[0238] outs, attns = [], []

[0239] for blk in self.boqs:

[0240] x_tokens, out, attn = blk(x_tokens, q)

[0241] outs.append(out)

[0242] attns.append(attn)

[0243] q = q + out # refine

[0244] out = torch.cat(outs, dim=1) # B, num_layers Q, C

[0245] out = self.fc(out.permute(0,2,1)) # B, C, row_dim

[0246] out = out.flatten(1)

[0247] out = F.normalize(out, p=2, dim=-1)

[0248] return out, attns

[0249] # -----------------------------

[0250] # GeM pooling

[0251] # -----------------------------

[0252] class GeM(nn.Module):

[0253] def __init__(self, p=3.0, eps=1e-6):

[0254] super().__init__()

[0255] self.p = nn.Parameter(torch.ones(1) p)

[0256] self.eps = eps

[0257] def forward(self, x):

[0258] return F.adaptive_avg_pool2d(x.clamp(min=self.eps).pow(self.p), (1,1)).pow(1. / self.p)

[0259] ```

[0260] The GeM and query_gen functions are used to obtain the image confidence value. Then, self-attention and cross-attention are used to aggregate the confidence value with the original features, and finally, one-dimensional features are obtained.

[0261] Furthermore, in the deep learning-based visual global location recognition and localization method proposed in this invention, step S3 is as follows:

[0262] Step S3.1: The high-dimensional descriptor is reduced in dimensionality using the feature quantization encoding method, and L2 normalization is used to ensure the consistent distribution of feature vectors, thereby improving the stability of similarity calculation.

[0263] The specific method involves first calculating the covariance matrix of the high-dimensional descriptor, then solving for its eigenvalues ​​and eigenvectors, and finally projecting the original descriptor onto the first few eigenvectors to obtain the dimensionality-reduced features. Then, L2 normalization is performed on the dimensionality-reduced eigenvectors, i.e., dividing each eigenvector by its Euclidean norm (the length of the vector). This ensures that the length of each eigenvector is 1, thus eliminating comparison bias caused by different feature sizes and making similarity calculation more stable.

[0264] Step S3.2: Compress the dimensionality-reduced features using vector quantization to reduce storage space and computational load for subsequent retrieval;

[0265] Step S3.3: Store the compressed feature vector and its corresponding image pose information together in the scene feature database for subsequent retrieval and localization calculations.

[0266] Furthermore, in the deep learning-based visual global location recognition and localization method proposed in this invention, step S4 is as follows:

[0267] Step S4.1: Obtain the image to be located and perform the same preprocessing operations as in step S1 (brightness normalization, distortion correction, and scale adjustment) on it to ensure that the input features are consistent with those in the training phase.

[0268] Similar to S1, the image to be located is preprocessed. First, brightness normalization is performed to adjust the brightness of the image to a fixed range (0 to 1) to eliminate the influence of illumination changes. Then, distortion correction is performed, and finally, the image size is unified.

[0269] Furthermore, in the deep learning-based visual global location recognition and localization method proposed in this invention, step S5 is as follows:

[0270] Step S5.1: Input the image to be located into the global feature extraction network to obtain the global feature descriptor corresponding to the image;

[0271] Consistent with S2, a one-dimensional feature descriptor of the image is obtained through image feature extraction and aggregation model.

[0272] Step S5.2: The extracted global feature descriptors are compared with the descriptors in the scene feature database. The similarity calculation method is cosine similarity. During the calculation, parallel calculation is performed by the graphics processor or central processing unit to accelerate the calculation and reduce the matching time.

[0273] The cosine similarity is calculated between the global feature descriptor of the image to be located and all feature descriptors in the scene feature database. The cosine similarity formula is:

[0274]

[0275] Where A and B represent the feature vectors of two different images. This represents the cosine similarity between two feature vectors A and B. During the calculation process, parallel computing is used by using a graphics processing unit or a central processing unit to accelerate the similarity calculation and reduce the calculation time. This can significantly improve efficiency, especially when the database is large.

[0276] Step S5.3: Sort the images in the database according to the similarity scores, and select several candidate matching frames with the highest scores for subsequent localization and matching.

[0277] Based on the calculated similarity scores, the images in the database are sorted, and the top N candidate matching frames (usually the top 5) with the highest scores are selected. These candidate frames will be used for subsequent localization and precise matching.

[0278] Furthermore, in the deep learning-based visual global location recognition and localization method proposed in this invention, step S6 is as follows:

[0279] Step S6.1: Extract local key point features from the image to be located and each candidate matching frame using the feature model, and establish corresponding point pairs between the two images using descriptor matching methods (such as nearest neighbor matching or cross-validation matching) to provide a basis for geometric verification.

[0280] The feature model can accurately detect salient keypoints in an image and extract their local descriptors. The nearest neighbor matching method is used to match the descriptors of the image to be located with those of candidate frames. During the matching process, the descriptor pair with the smallest distance is selected as the candidate matching point.

[0281] Step S6.2: Use the random consistency sampling algorithm to fit a geometric model in the feature point matching pairs between the candidate matching frame and the image to be located. During the iteration process, outlier matching points that do not conform to the geometric model are automatically removed to improve the robustness of matching and determine the interior point set of the optimal geometric model, so as to provide a reliable matching basis for subsequent reprojection error calculation.

[0282] In the feature point matching task between candidate matching frames and the image to be located, a method combining random consistency sampling algorithm with specific geometric model verification is adopted to improve the robustness of matching. First, an appropriate geometric model needs to be selected according to the application scenario. If the matching objects are the same plane (such as a wall or tabletop), the homography matrix (H) is used to describe the projection transformation relationship under different perspectives (only 4 pairs of non-collinear matching points are needed for fitting). Then, the random consistency sampling algorithm iterative process is entered: each time, the minimum number of matching points (4 pairs) are randomly selected from all feature point matching pairs, and the corresponding geometric model is fitted based on these sampling points. Then, the reprojection error of all remaining matching points to the model is calculated, and matching points with errors less than a preset threshold are judged as inliers, and the number of inliers is counted. The above process of "random sampling - fitting model - counting inliers" is repeated until the preset number of iterations. Finally, the model with the most inliers (or the highest proportion of inliers) is selected as the optimal geometric model. The corresponding set of inliers is the reliable matching pair after removing outliers and mismatches, providing an accurate and interference-free matching basis for subsequent reprojection error calculation, and effectively avoiding the interference of outliers on positioning accuracy.

[0283] Step S6.3: Using the fitted geometric model, project the key points of the image to be localized onto the candidate frames and calculate the reprojection error. Set an error threshold and remove matching frames or matching point pairs whose reprojection errors exceed the threshold. The remaining matching frames constitute the final set of valid matching frames for subsequent accurate pose estimation.

[0284] First, candidate frames are screened. Specifically, the proportion of inliers after feature point matching between the candidate frame and the frame to be matched is compared to whether it is greater than 50%. If it is not, the candidate frame is removed. Finally, the set of valid candidate frames and feature point matching relationships are obtained.

[0285] Furthermore, in the deep learning-based visual global location recognition and localization method proposed in this invention, step S7 is as follows:

[0286] Step S7.1: Based on the correspondence of key points extracted from the set of valid matching frames, construct the pose optimization problem of the image to be located, take the camera pose parameters (rotation matrix and translation vector) as the variables to be optimized, and take the reprojection error of the matching points as the objective function to ensure that the optimization process can consider the geometric constraints of all valid matching frames at the same time.

[0287] The specific optimization variables include the pose of the image to be localized. , (Set as frame 0) and the coordinates of all valid matching points on the 3D map. Then, based on the observation constraints, a reprojection error is constructed for each valid matching pair (the pixel coordinates of the k-th frame). With 3D map point coordinates ), there are observation constraints: ,in Let `proj` be the intrinsic parameter matrix of the camera in frame k (if the same camera is used for all frames). `proj(·)` is the projection function, which projects points in 3D space through the camera's intrinsic and extrinsic parameters (rotation matrix). With translation vector This transforms the image onto a two-dimensional image plane, simulating the camera imaging process. Specifically, the function first uses extrinsic parameters... and 3D map points Transform from world coordinate system to camera coordinate system, and then use the intrinsic parameter matrix. The points in the camera coordinate system are projected onto the image plane to obtain the corresponding pixel coordinates.

[0288] Step S7.2: Use a global optimization algorithm based on nonlinear least squares (such as Levenburg-Marquardt algorithm or Gauss-Newton algorithm) to iteratively solve for the optimal camera pose. By continuously adjusting the rotation and translation parameters, minimize the reprojection error of all matching points, thereby improving the accuracy and robustness of pose estimation.

[0289] Specifically, we first construct the reprojection error objective function, as shown in the following formula:

[0290]

[0291] in Let be the set of valid interior points of the frame to be located (let's say it's the j-th frame). The weight of the i-th matching pair in the j-th frame is usually set to 1 / , The standard deviation of the reprojection error for this matching pair. This represents the reprojection error of the i-th matching pair in the j-th frame. Let j be the direction and position of the j-th frame. Represents the coordinates of a 3D point in three-dimensional space. This means that the minimum value of the expression is required, while the poses of other frames remain fixed and do not participate in the optimization.

[0292] After constructing the objective function, it is solved using the Levenberg-Marquardt least squares optimization method. The specific steps are as follows:

[0293] 1. Error function linearization. For each observation error... Perform a first-order Taylor expansion at the initial estimate.

[0294] 2. Construct a system of equations. Substitute the linearized error into the objective function, then differentiate the quantity to be solved, set the derivative to 0, and thus solve for the increment of the value to be optimized.

[0295] 3. Add the increment to the initial value to be optimized to obtain a new estimate. Then, recalculate the objective function based on the new estimate. Repeat the above steps until the objective function is less than a set threshold (generally, the threshold is set to a certain value). If the iteration stops, then stop.

[0296] Step S7.3: After optimization, the obtained camera rotation matrix and translation vector are converted into three-dimensional position and pose information output to provide accurate camera pose results for subsequent localization, navigation or scene understanding tasks.

[0297] This invention provides a deep learning-based method for visual global position recognition and localization, aiming to solve the problems of low localization accuracy, poor robustness to changes in illumination and viewing angle, and insufficient real-time performance of existing visual localization methods in complex environments. The method includes the following steps: First, the acquired image is preprocessed to eliminate distortion and illumination effects; then, the preprocessed image is input into a deep learning network to extract highly discriminative global feature descriptors, and feature aggregation and dimensionality reduction methods are used to reduce storage and computational overhead; next, the features are quickly matched with features in a scene database using similarity, and erroneous matches are eliminated using geometric consistency verification; finally, based on the verified matching results, pose optimization is performed using a nonlinear least squares method to output the three-dimensional position and pose information of the image to be localized. This invention has the advantages of high computational efficiency, high localization accuracy, and strong robustness, and is suitable for real-time localization and navigation tasks in complex indoor and outdoor environments.

[0298] Experimental objective: To design a complete comparative experiment for the method of this invention, and to quantitatively analyze its performance advantages over traditional methods.

[0299] Experimental environment: Hardware: Ubuntu 20.04 + 1080ti GPU; Software: Traditional positioning method vs. the positioning method of this invention.

[0300] Experimental Methods: Localization tasks were performed using both traditional and the proposed method in scenarios with publicly available and self-built datasets. The publicly available datasets provide rich multi-view images and ground truth pose information, which can be used to compare the localization accuracy of the two methods. The self-built datasets simulate more complex and diverse environmental scenarios, further verifying the robustness of the proposed method under different lighting, weather, and viewpoint changes. In the experiments, localization was first performed using both methods, and the pose estimation results for each method were obtained. For each frame, the localization method estimated the camera pose (including rotation matrix and translation vector) through steps such as feature extraction, matching, and geometric consistency verification. The localization errors of the two methods were evaluated by comparing them with the ground truth results provided by the datasets. The experimental results are shown in the table below:

[0301] Table 1. Comparison of performance indicators between traditional methods and the method of this invention

[0302]

[0303] Experimental results show that the proposed method outperforms traditional methods on multiple public datasets and self-built complex scene datasets. Under different lighting, weather, and viewpoint conditions, the method significantly improves localization recall (average recall increased by 10%) and localization accuracy (improved by 20%), while maintaining high accuracy and achieving real-time inference, with a single-frame processing speed improvement of approximately 40%. Furthermore, through feature aggregation and dimensionality reduction design, the computational overhead of feature storage and retrieval is effectively reduced, enabling the system to be deployed on embedded platforms with limited computing resources, demonstrating strong practical value and potential for widespread application.

[0304] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A deep learning-based visual global position recognition and positioning method, characterized in that, The method comprises the following steps: S1, collecting a multi-view image sequence of a scene to be mapped, and preprocessing the collected images; S2, inputting the preprocessed images into a global feature extraction network, obtaining a global feature descriptor corresponding to each image through multi-layer feature extraction and global aggregation operation; S3, dimensionally reducing, normalizing and compressing the global feature descriptor, and storing the processed descriptor and the corresponding image pose information in a scene feature database; S4, obtaining an image to be positioned, and performing the same preprocessing operation as step S1 on the image to be positioned; S5, inputting the image to be positioned into the global feature extraction network in step S2 to obtain a global feature descriptor of the image to be positioned, and performing similarity calculation on the descriptor with the descriptors in the scene feature database, and screening a plurality of candidate matching frames according to the similarity scores; S6, performing geometric consistency verification on the candidate matching frames to eliminate incorrect matches that do not satisfy geometric constraints, and obtaining a final set of effective matching frames; S7, solving the camera pose of the image to be positioned by using a pose optimization algorithm according to the set of effective matching frames, and outputting the three-dimensional position and attitude information of the image to be positioned. 2.The method of claim 1, wherein, Step S1 specifically comprises the following steps: S11, collecting a multi-view image sequence of a scene to be mapped; S12, eliminating light intensity and color deviation by brightness normalization and color equalization algorithm; S13, correcting the image according to the camera calibration parameters, and scaling the image to a unified resolution. 3.The method of claim 1, wherein, Step S2 specifically comprises the following steps: S21, inputting the preprocessed image into a constructed deep neural network model, extracting multi-scale image feature representation through multi-layer convolution and feature encoding; S22, performing global pooling and attention aggregation on the extracted features, compressing the spatial dimension into a single vector representation, and obtaining the global feature descriptor of the image. 4.The method of claim 1, wherein, Step S3 specifically comprises the following steps: S31, dimensionally reducing the high-dimensional descriptor by using a feature quantization encoding method, and performing L2 normalization on the dimensionally reduced feature vector; S32, compressing the dimensionally reduced feature by using vector quantization; S33, storing the compressed feature vector and the corresponding image pose information in the scene feature database. 5.The method of claim 1, wherein, Step S5 specifically comprises the following steps: S51, inputting the image to be positioned into the global feature extraction network to obtain a global feature descriptor corresponding to the image to be positioned; S52, calculating the similarity of the extracted global feature descriptor and the descriptors in the scene feature database using cosine similarity, and performing parallel acceleration calculation through a graphics processing unit or a central processing unit during the calculation process; S53, sorting the images in the database according to the similarity scores, and selecting a plurality of candidate matching frames with the highest scores. 6.The method of claim 1, wherein, Step S6 specifically comprises the following steps: S61, extracting local key point features from the image to be positioned and each candidate matching frame through a feature model, and establishing corresponding point pairs between the two images through a descriptor matching method; S62, fitting a geometric model in the candidate matching frame and the feature point matching pairs of the image to be positioned by using a random consistency sampling algorithm, automatically eliminating outlier matching points that do not conform to the geometric model in the iteration process; S63, projecting the key points of the image to be positioned into the candidate frame using the fitted geometric model, calculating the re-projection error, setting an error threshold, eliminating the matching frame or matching point pair whose re-projection error exceeds the threshold, and obtaining the final effective matching frame set composed of the remaining matching frames. 7.The method of claim 1, wherein, Step S7 specifically includes the following steps: S71, constructing a pose optimization problem of the image to be positioned according to the correspondence relationship of the key points extracted in the effective matching frame set, taking the camera pose parameters as the variables to be optimized, and taking the re-projection error of the matching points as the optimization objective function; S72, iteratively solving the optimal camera pose by using a global optimization algorithm based on a nonlinear least square method, and minimizing the re-projection error of all matching points by continuously adjusting the rotation and translation parameters; S73, after optimization, converting the solved camera rotation matrix and translation vector into three-dimensional position and attitude information and outputting. 8.The method of claim 7, wherein, The specific formula of the objective function is: ； wherein is a set of valid inliers of the jth frame to be localized, is a weight of the ith matching pair of the jth frame, usually set as 1 , is a standard deviation of re-projection error of the matching pair, denotes a re-projection error of the ith matching pair of the jth frame, is a direction and position of the jth frame, denotes a coordinate of a 3D point in a three-dimensional space, denotes a requirement for the minimum value of the formula. 9.The method of claim 8, wherein, Step S72 specifically includes the following steps: S721, linearize the error function for each observation error First order Taylor expansion at the initial estimate; S722, substituting the linearized error into the objective function, then deriving the quantity to be solved, and setting the derivative to 0 to solve the increment of the value to be optimized; S723, adding the increment to the initial value to be optimized to obtain a new estimated value, then recalculating the objective function according to the new estimated value, repeating the above steps, and stopping the iteration until the objective function is less than the set threshold.

Images