Multi-scale visual positioning methods, devices, storage media, and electronic devices

Abstract

This application provides a multi-scale visual positioning method, device, storage medium, and electronic device. The method includes: acquiring 2D image data and point cloud data within a monitoring range; extracting features from the 2D image data using a ResNet network to obtain 2D image features; extracting features from the point cloud data using a KPFCNN network to obtain point cloud features; performing feature matching between the 2D image features and point cloud features using a Transformer network to obtain feature matching results; and determining the three-dimensional coordinates of the target object based on the feature matching results. This method utilizes deep learning to automatically learn effective features from images and point clouds, and refines these features using a Transformer network to improve their discriminative power and robustness, making them more resistant to the influence of factors such as illumination and occlusion.

Description

Technical Field

[0001] This application relates to the field of visual positioning, and more specifically, to a multi-scale visual positioning method, a multi-scale visual positioning device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Traditional 2D-3D correspondence visual localization methods are mainly divided into two categories:

[0003] 1. Feature-matching-based methods extract feature points from images and point clouds and match them to establish a 2D-3D correspondence. The PnP method of SIFT / ORB feature matching is a typical example. These methods mainly rely on manually designed feature descriptors. For example, SIFT and ORB algorithms each design unique feature descriptors to describe local features in the image. This design process is time-consuming and laborious, and it is difficult to guarantee the applicability of feature descriptors to different scenes and targets. In addition, matching strategies need to be designed, such as nearest neighbor matching or brute-force matching, to match image feature points with point cloud feature points. These strategies are easily affected by noise, occlusion, and other factors, leading to matching errors, and are difficult to adapt to complex scenes and large-scale data.

[0004] 2. Model-matching methods establish a 2D-3D correspondence by matching a 3D model with a point cloud. The ICP algorithm is a typical example. These methods require a pre-built 3D model, which limits their applicability. They cannot be used for targets where a 3D model is unavailable. Furthermore, model-matching methods struggle to adapt to dynamic changes in the scene, such as changes in target pose or occlusion, leading to decreased matching accuracy.

[0005] Therefore, traditional 2D-3D correspondence visual positioning methods have low matching accuracy, poor robustness, and cannot effectively solve the scale ambiguity problem caused by perspective projection. Summary of the Invention

[0006] The main objective of this application is to provide a multi-scale visual positioning method, a multi-scale visual positioning device, a computer-readable storage medium, and an electronic device, so as to at least solve the problems of low matching accuracy, poor robustness, and inability to effectively resolve scale ambiguity caused by perspective projection in the existing 2D-3D correspondence visual positioning methods.

[0007] To achieve the above objectives, according to one aspect of this application, a multi-scale visual positioning method is provided, comprising: acquiring 2D image data and point cloud data within a monitoring range; extracting features from the 2D image data using a ResNet network to obtain 2D image features; extracting features from the point cloud data using a KPFCNN network to obtain point cloud features; performing feature matching between the 2D image features and the point cloud features using a Transformer network to obtain feature matching results, and determining the three-dimensional coordinates of a target object based on the feature matching results.

[0008] Optionally, a Transformer network is used to perform feature matching on the image features and the point cloud features to obtain a feature matching result, including: enhancing the image features to obtain enhanced image features, and enhancing the point cloud features to obtain enhanced point cloud features; determining an anchor feature matrix based on the enhanced image features, wherein the elements in the anchor feature matrix are the enhanced image features; determining a memory feature matrix based on the enhanced point cloud features, wherein the elements in the memory feature matrix are the enhanced point cloud features; and using a Transformer network with an attention mechanism to model the pairwise association between the elements in the anchor feature matrix and the elements in the memory feature matrix to obtain the feature matching result.

[0009] Optionally, the image features are enhanced to obtain enhanced image features, and the point cloud features are enhanced to obtain enhanced point cloud features, including: obtaining a first location information enhancement formula. in, These are 2D image features enhanced with location information. For the 2D image features, Let φ(·) be the pixel coordinate matrix of the downsampled image, and let φ(·) be the Fourier embedding function; obtain the formula for enhancing the second location information. in, The point cloud features are augmented with location information. For the point cloud features, The points are the points corresponding to the coarsest layer in the KPFCNN network; the image features are enhanced using the first location information enhancement formula to obtain the enhanced image features; the point cloud features are enhanced using the second location information enhancement formula to obtain the enhanced point cloud features.

[0010] Optionally, a Transformer network is used to model the pairwise associations between elements in the anchor feature matrix and elements in the memory feature matrix through an attention mechanism to obtain the feature matching result. This includes: obtaining a first projection weight, a second projection weight, and a third projection weight, where the first projection weight is the projection weight of the query condition, the second projection weight is the projection weight of the feature key, and the third projection weight is the projection weight of the feature value; projecting the anchor feature matrix and the memory feature matrix using the first projection weight, the second projection weight, and the third projection weight to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix; and inputting the correlation into a shallow MLP network to further project attention features to obtain the feature matching result.

[0011] Optionally, the anchor feature matrix and the memory feature matrix are projected using the first projection weight, the second projection weight, and the third projection weight to obtain the relevance between each element in the anchor feature matrix and each element in the memory feature matrix, including: obtaining the query condition calculation formula Q = F. A W Q The formula for calculating characteristic keys is K = F. M W K The formula for calculating eigenvalues ​​is V = F. M W V Where Q is the query condition, K is the feature key, V is the feature value, and F is the feature value. A F is an element in the anchor point feature matrix. M For the elements in the memory feature matrix, W Q W is the first projection weight. K W is the second projection weight. V The third projection weights are used to obtain the attention feature formula. Among them, Attention(F A F M ) is F A and F M The relevance between the elements is determined by the following formula: Q is the query condition, K is the feature key, V is the feature value, and d is the number of columns in the anchor feature matrix and the memory feature matrix. The relevance between each element in the anchor feature matrix and each element in the memory feature matrix is ​​determined according to the attention feature formula.

[0012] Optionally, obtaining the first projection weights, the second projection weights, and the third projection weights includes: obtaining initial first weights, initial second weights, and initial third weights; obtaining the model total loss formula L. all =L coarse +λL fine , where Lall L represents the total loss of the model. coarse Let L be the first circumferential loss, λ be the balance factor, and L be the circumferential loss. fine The second circular loss is related to the initial first weight, the initial second weight, and the initial third weight. The first circular loss is a circular loss calculated based on the patch feature distance at a coarse level, where the coarse level is defined by the overlap ratio of positive and negative samples based on the patch feature distance. The second circular loss is a circular loss calculated based on the feature distance between pixels and points at a fine level, where the fine level is defined by the spatial distance between positive and negative samples based on the spatial distance between pixels and points. According to the formula for the total model loss, the total model loss is determined, and the initial first weight, the initial second weight, and the initial third weight are adjusted to minimize the total model loss. The initial first weight, the initial second weight, and the initial third weight corresponding to the minimum total model loss are determined as the first projection weight, the second projection weight, and the third projection weight.

[0013] Optionally, before obtaining the total model loss formula, the method further includes: obtaining the circumferential loss formula. Where, d i For anchor descriptors, L i For d i The corresponding circumferential loss value, one anchor descriptor corresponds to one circumferential loss value. It is the Euclidean distance between the i-th sample and the j-th sample. and These are the individual weights of positive and negative sample pairs, while and It is the ratio factor between positive and negative sample pairs, Δ p Δ n It is the expected distance between positive samples and the expected distance between negative samples; the sum of all the said circumferential loss values ​​is determined as the circumferential loss.

[0014] According to another aspect of this application, a multi-scale visual positioning device is provided, comprising: an acquisition unit for acquiring 2D image data and point cloud data within a monitoring range; a first extraction unit for extracting features from the 2D image data using a ResNet network to obtain 2D image features; a second extraction unit for extracting features from the point cloud data using a KPFCNN network to obtain point cloud features; and a determination unit for performing feature matching on the 2D image features and the point cloud features using a Transformer network to obtain a feature matching result, and determining the three-dimensional coordinates of a target object based on the feature matching result.

[0015] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the described multi-scale visual positioning methods.

[0016] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any of the described multi-scale visual localization methods.

[0017] Applying the technical solution of this application, the aforementioned multi-scale visual positioning method first acquires 2D image data and point cloud data within the monitoring range; then, it uses a ResNet network to extract features from the 2D image data, obtaining 2D image features; next, it uses a KPFCNN network to extract features from the point cloud data, obtaining point cloud features; finally, it uses a Transformer network to perform feature matching on the 2D image features and point cloud features, obtaining feature matching results, and determining the three-dimensional coordinates of the target object based on the feature matching results. This method utilizes deep learning methods to automatically learn effective features in images and point clouds, and refines these features through a Transformer network, improving the discriminative power and robustness of the features, making them more resistant to the influence of factors such as illumination and occlusion. It solves the problems of low matching accuracy, poor robustness, and inability to effectively resolve scale ambiguities caused by perspective projection in existing 2D-3D correspondence visual positioning methods. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 A hardware structure block diagram of a mobile terminal performing a multi-scale visual positioning method according to an embodiment of this application is shown.

[0020] Figure 2 A flowchart illustrating a multi-scale visual positioning method according to an embodiment of this application is shown.

[0021] Figure 3 A network architecture diagram of a Transformer network according to an embodiment of this application is shown;

[0022] Figure 4 A flowchart illustrating a multi-scale visual positioning method according to an embodiment of this application is shown.

[0023] Figure 5 A structural block diagram of a multi-scale visual positioning device provided according to an embodiment of this application is shown.

[0024] The above figures include the following reference numerals:

[0025] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0030] 2D-3D correspondence: refers to the mapping relationship between pixels in an image and corresponding points in a 3D point cloud.

[0031] Transformer: A deep learning model that learns the relationships within data through a self-attention mechanism.

[0032] ResNet: A deep convolutional neural network that solves the vanishing gradient problem in deep network training through residual connections.

[0033] FPN: A feature pyramid network that fuses feature maps from different levels to obtain feature maps of different scales.

[0034] KPFCNN: A point cloud learning network based on KPConv, where KPConv is a convolutional operation for point clouds.

[0035] Multi-scale sampling: Segmenting the image into blocks of different scales to alleviate matching ambiguity.

[0036] Block pyramid: Divide each image block into different levels to extract more fine-grained features.

[0037] As introduced in the background section, deep learning methods have made significant progress in image recognition and 3D point cloud processing in recent years, providing new approaches for visual localization of 2D-3D correspondences. For example, deep learning-based feature extraction methods can automatically learn features in images and point clouds without the need for manual feature descriptor design; deep learning-based feature matching methods can utilize neural networks to learn the similarity between features, improving matching robustness; and deep learning-based model matching methods can learn the parameters of 3D models without the need for pre-building 3D models.

[0038] However, existing 2D-3D correspondence visual positioning methods have certain technical defects, which limit their application in complex scenes and large-scale data.

[0039] Manually designed feature descriptors and matching strategies are difficult to adapt to complex scenarios and large-scale data.

[0040] Traditional feature descriptors, such as SIFT and ORB, require manual design, a time-consuming and labor-intensive process with limited applicability. Different scenarios and targets may require different feature descriptors; for example, SIFT is more suitable for scenes with rich textures, while ORB is more suitable for scenes with fewer textures. This design process not only requires a significant amount of time and expertise but also makes it difficult to guarantee the applicability of feature descriptors to different scenarios and targets.

[0041] Traditional matching strategies, such as nearest neighbor matching or brute-force matching, are easily affected by noise, occlusion, and other factors, leading to matching errors. For example, in complex scenarios with varying lighting, occlusion, and background interference, the difficulty of feature point extraction and matching increases significantly, further reducing matching robustness. Furthermore, these matching strategies are ill-suited for processing large-scale data, limiting their efficiency in practical applications. Moreover, the feature extraction and matching processes requiring manual intervention are ill-suited for large-scale data processing. For instance, in applications such as autonomous driving and robot navigation, where massive amounts of image and point cloud data need to be processed, the efficiency of traditional methods is insufficient to meet practical needs.

[0042] Meanwhile, changes in lighting can alter the brightness and contrast of feature points in images and point clouds, affecting the accuracy of feature point extraction and matching. For example, under direct sunlight or backlighting, the characteristics of feature points may change significantly, leading to matching failures. Occlusion can prevent some feature points from being detected, thus affecting matching accuracy. For instance, occlusion is common in scenes with buildings and trees, reducing the robustness of traditional matching methods. Background interference can cause mismatches in feature point extraction and matching, affecting localization accuracy. For example, in scenes with complex backgrounds, such as city streets, the textures and colors in the background may be similar to the target object, leading to mismatches.

[0043] Furthermore, existing technologies cannot effectively resolve the scale ambiguity problem caused by perspective projection. Perspective projection causes objects in an image to appear larger in the foreground and smaller in the background. Traditional 2D-3D correspondence-based visual localization methods struggle to address this issue effectively, leading to decreased matching accuracy. Scale ambiguity can cause matching errors, such as mismatching distant objects with nearby objects, or vice versa.

[0044] Existing 2D-3D correspondence visual localization methods suffer from technical defects such as manually designed feature descriptors and matching strategies, poor matching robustness, and inability to effectively resolve scale ambiguity caused by perspective projection, making them difficult to adapt to complex scenes and large-scale data applications.

[0045] To address the problems of low matching accuracy, poor robustness, and inability to effectively resolve scale ambiguity caused by perspective projection in existing 2D-3D correspondence visual positioning methods, embodiments of this application provide a multi-scale visual positioning method, a multi-scale visual positioning device, a computer-readable storage medium, and an electronic device.

[0046] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0047] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a multi-scale visual positioning method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0048] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multi-scale visual positioning method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] This embodiment provides a multi-scale visual positioning method that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0050] Figure 2 This is a flowchart of a multi-scale visual positioning method according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0051] Step S201: Obtain 2D image data and point cloud data within the monitoring range;

[0052] Specifically, in the data preparation phase, 2D images and prior point cloud data of the monitored scene are first collected. Image data can be captured by surveillance cameras and preprocessed, such as cropping unnecessary parts and adjusting image size. Point cloud data can be scanned by devices such as LiDAR and preprocessed, such as removing noise and downsampling. To improve the model's generalization ability, the images and point cloud data are labeled, including labeling the location and category of the monitored objects.

[0053] Step S202: Use a ResNet network to extract features from the 2D image data to obtain 2D image features;

[0054] Specifically, given a pair of images and point clouds, hierarchical feature extraction is performed using two modality-specific encoder-decoder backbone networks. For images, this invention uses ResNet and Feature Pyramid Networks (FPN) to generate multi-scale image features. Given Q∈R^(H×W×2) as the original image pixel coordinate matrix and Q^∈R^(H×W×2) as the downsampled image pixel coordinate matrix, coarse- and fine-level matching is performed between the minimum resolution downsampled 2D features and the original resolution features.

[0055] Step S203: Use the KPFCNN network to extract features from the point cloud data to obtain point cloud features;

[0056] Specifically, for point clouds, we employ KPFCNN to learn 3D features. Unlike images with a fixed resolution, point clouds typically have inconsistent sizes, and KPFCNN dynamically downsamples them using grid downsampling. ResNet and FPN are responsible for extracting image features, while KPFCNN is responsible for extracting point cloud features. We use the points P^∈R^(N^×3) corresponding to the coarsest layer and their associated features F^^p∈R^(N^×C^).

[0057] To obtain richer feature information, the above steps use ResNet and KPFCNN networks to extract features from the image and point cloud, respectively. ResNet effectively extracts deep features from the image and constructs a feature pyramid using an FPN structure to obtain feature maps at different scales. KPFCNN effectively extracts local features from the point cloud and performs feature extraction and propagation through KPConv convolution operations, thereby obtaining richer point cloud feature information.

[0058] Step S204: Use a Transformer network to perform feature matching on the 2D image features and the point cloud features to obtain feature matching results, and determine the three-dimensional coordinates of the target object based on the feature matching results.

[0059] Specifically, these features are fed into the Transformer network for feature refinement to obtain richer feature information. Multi-scale feature maps extracted by ResNet, FPN, and KPFCNN are used to resolve scale ambiguity caused by perspective projection. The Transformer network further improves matching accuracy by learning the correlation between different scales.

[0060] In the real-time localization phase, the above embodiments input the 2D image of the real-time monitoring scene and the prior point cloud into the trained model. The model automatically extracts features from the image and point cloud, refines and matches these features, and finally obtains the correspondence between pixels and the point cloud. Image detection and segmentation techniques are used to segment the person in the image, obtaining the region where the person is located. The matching framework of this invention is used to calculate the correspondence between each pixel in the segmented region and the point cloud points, mapping each pixel in the segmented region of the image to the point cloud, thereby obtaining the position coordinates of the person in 3D space.

[0061] In the image branch, besides ResNet, other deep convolutional neural networks such as VGG, Inception, and DenseNet can be used to extract image features. In the point cloud branch, besides KPFCNN, point cloud processing networks such as PointNet, PointNet++, and DeepGCN can be used to extract point cloud features. For Transformer feature refinement, in addition to using the standard Transformer structure, multi-head self-attention mechanisms, non-local neural networks, or other self-attention-based models can be considered for feature fusion and refinement. These alternatives can all achieve the objective of this invention, namely, improving the accuracy and robustness of 2D-3D correspondence visual localization.

[0062] The above steps refine the matching process step by step, extracting the pixel-to-pixel correspondence between the image and the point cloud without requiring keypoint detection between the two modalities. A multi-scale block pyramid is constructed to alleviate scale ambiguity in the block matching process. This coarse-to-fine matching process gradually eliminates erroneous matches, ultimately yielding a more accurate 2D-3D correspondence.

[0063] Specifically, a Transformer network is used to perform feature matching on the image features and the point cloud features to obtain feature matching results, including:

[0064] Step S301: Enhance the image features to obtain enhanced image features, and enhance the point cloud features to obtain enhanced point cloud features;

[0065] The process of enhancing the image features to obtain enhanced image features and enhancing the point cloud features to obtain enhanced point cloud features includes the following steps:

[0066] Step S3011: Obtain the first position information enhancement formula in, These are 2D image features enhanced with location information. For the 2D image features, Let φ(·) be the pixel coordinate matrix of the downsampled image, and let φ(·) be the Fourier embedding function.

[0067] Step S3012: Obtain the second position information enhancement formula in, The point cloud features are augmented with location information. For the point cloud features, These are the points corresponding to the coarsest layer in the KPFCNN network;

[0068] Step S3013: The image features are enhanced using the first location information enhancement formula to obtain the enhanced image features;

[0069] Step S3014: The point cloud features are enhanced using the second location information enhancement formula to obtain the enhanced point cloud features.

[0070] Specifically, Where L is the embedding length. To simplify computation, the first two spatial dimensions (height and width) of the 2D feature matrix are flattened into one dimension. Used for future calculations. These are 2D image features and 3D point cloud features enhanced with location information.

[0071] Among these methods, enhancing image features and point cloud features can yield more accurate image features and point cloud features.

[0072] Step S302: Determine the anchor feature matrix based on the enhanced image features, wherein the elements in the anchor feature matrix are the enhanced image features;

[0073] Step S303: Determine a memory feature matrix based on the enhanced point cloud features, wherein the elements in the memory feature matrix are the enhanced point cloud features;

[0074] Step S304: Using a Transformer network and an attention mechanism, the elements in the anchor feature matrix and the elements in the memory feature matrix are modeled to be associated pairwise, thereby obtaining the feature matching result.

[0075] Specifically, the features are further refined using a transformer in two ways. Given the anchor feature matrix F... A ∈R |A|×d and memory feature matrix F M ∈R |M|×d The transformer models the pairwise relationships between them through an attention mechanism, generating more discriminative features.

[0076] Feature refinement is achieved using Transformer networks. Transformer networks effectively capture long-range dependencies and learn the correlations between different modalities through self-attention and cross-attention mechanisms, thereby extracting richer feature information. This can further improve the discriminative power and robustness of features.

[0077] The Transformer network is used for feature refinement. By learning the contextual information and cross-modal correlation within the image and point cloud through self-attention and cross-attention mechanisms, the discriminative power and robustness of the features are improved, thereby enhancing the matching accuracy and robustness.

[0078] Specifically, the Transformer network, through an attention mechanism, models the pairwise association between elements in the anchor feature matrix and elements in the memory feature matrix to obtain the feature matching result, including the following steps:

[0079] Step S401: Obtain the first projection weight, the second projection weight, and the third projection weight. The first projection weight is the projection weight of the query condition, the second projection weight is the projection weight of the feature key, and the third projection weight is the projection weight of the feature value.

[0080] The process of obtaining the first projection weight, the second projection weight, and the third projection weight includes the following steps:

[0081] Step S4011: Obtain the initial first weight, initial second weight, and initial third weight;

[0082] Step S4012, obtain the total loss formula L of the model. all =L coarse +λL fine , where L all L represents the total loss of the model. coarse Let L be the first circumferential loss, λ be the balance factor, and L be the circumferential loss. fineThe second circular loss is related to the initial first weight, the initial second weight, and the initial third weight. The first circular loss is a circular loss calculated based on the patch feature distance at a coarse level, where the coarse level is defined by the overlap ratio of positive and negative samples based on the patch feature distance. The second circular loss is a circular loss calculated based on the feature distance between pixels and points at a fine level, where the fine level is defined by the spatial distance between positive and negative samples based on the pixel and point feature distance.

[0083] Step S4013: Determine the total model loss according to the total model loss formula, and adjust the initial first weight, the initial second weight, and the initial third weight to minimize the total model loss. Then, determine the initial first weight, the initial second weight, and the initial third weight corresponding to the minimum total model loss as the first projection weight, the second projection weight, and the third projection weight.

[0084] Specifically, compared to contrastive loss and triple loss, circular loss has a circular decision boundary, which is beneficial for convergence. Given an anchor descriptor d... i Its positive and negative pairs are represented as and

[0085] Before obtaining the formula for the total loss of the model, the method further includes the following steps:

[0086] Step S501: Obtain the circumferential loss formula Where, d i For anchor descriptors, L i For d i The corresponding circumferential loss value, one anchor descriptor corresponds to one circumferential loss value, It is the Euclidean distance between the i-th sample and the j-th sample. and These are the individual weights of positive and negative sample pairs, while and It is the ratio factor between positive and negative sample pairs, Δ p Δ n These are the expected distances between positive samples and the expected distances between negative samples;

[0087] Step S502: The sum of all the circumferential loss values ​​is determined as the circumferential loss.

[0088] Specifically, the model is trained using metric learning. The model architecture is as follows: Figure 3As shown. At the coarse level, a scaled circular loss is used to supervise patch features. At the fine level, a different standard circular loss is used to supervise dense pixel and point features. The total loss is then calculated.

[0089] Step S402: Project the anchor feature matrix and the memory feature matrix using the first projection weight, the second projection weight, and the third projection weight to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix;

[0090] The process of projecting the anchor feature matrix and the memory feature matrix using the first projection weight, the second projection weight, and the third projection weight to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix includes the following steps:

[0091] Step S4021: Obtain the query condition calculation formula Q = F A W Q The formula for calculating characteristic keys is K = F. M W K The formula for calculating eigenvalues ​​is V = F. M W V Where Q is the query condition, K is the feature key, V is the feature value, and F is the feature value. A F is an element in the anchor point feature matrix. M For the elements in the memory feature matrix, W Q W is the first projection weight. K W is the second projection weight. V The third projection weight;

[0092] Step S4022: Obtain the attention feature formula Among them, Attention(F A F M ) is F A and F M The relevance between them, Q is the query condition, K is the feature key, V is the feature value, d is...;

[0093] Step S4023: Determine the correlation between each element in the anchor feature matrix and each element in the memory feature matrix according to the attention feature formula.

[0094] Specifically, during the model training phase, a Transformer-based multi-scale feature matching model is trained. Among them, W... Q W K W V ∈R d×dThe model, which uses projection weights for Query, Key, and Value, includes image and point cloud branches to extract features from the image and point cloud, respectively. To improve feature discriminativeness and robustness, this invention utilizes a Transformer network for feature refinement, learning the internal contextual information and cross-modal correlations of the image and point cloud through self-attention and cross-attention mechanisms. To address the scale ambiguity problem caused by perspective projection, this invention designs a multi-scale sampling and matching strategy based on an image spatial pyramid. By segmenting the image into blocks of different scales and matching them with point cloud features, matching ambiguity can be effectively alleviated and learning difficulty reduced, thereby improving matching accuracy and robustness. Furthermore, a coarse-to-fine matching process is adopted, progressively refining the matching process without requiring keypoint detection between the two modalities. Constructing a multi-scale block pyramid further alleviates scale ambiguity in the block matching process, thereby improving matching accuracy. To improve computational efficiency, the Adam optimization algorithm is used to train the model, and cross-validation and other methods are used to evaluate and fine-tune the model performance.

[0095] Step S403: Input the relevance into a shallow MLP network to further project attention features and obtain the feature matching result.

[0096] Specifically, a shallow MLP is used to further project attention features as the final output features. The best match is found by comparing image patches and 3D patches at different scales. A coarse block correspondence is extracted using a mutual top-k selection method. These correspondences represent the initial match between image patches and 3D patches.

[0097] For each coarse correspondence, features of local pixels and points at a fine level are collected from the image and point cloud. The collected 2D and 3D features are normalized into unit-length vectors for easy comparison.

[0098] The above steps then use a mutual top-k selection method to match pixels and points, generating locally dense correspondences. This method is more efficient than using a specific matching layer (such as Sinkhorn), especially when the 2D blocks are large. All local correspondences are then aggregated to form the final set of dense pixel-point correspondences.

[0099] In some instances, such as Figure 4As shown, given a pair of images and point clouds, hierarchical feature extraction is performed using two modality-specific encoder-decoder backbone networks. For images, this invention uses ResNet and Feature Pyramid Networks (FPN) to generate multi-scale image features. Given Q∈R^(H×W×2) as the original image pixel coordinate matrix and Q^∈R^(H×W×2) as the downsampled image pixel coordinate matrix, the minimum resolution downsampled 2D features are matched with the original resolution at coarse and fine levels. For point clouds, we use KPFCNN to learn 3D features. Unlike images with a fixed resolution, point clouds typically have inconsistent sizes, and KPFCNN dynamically downsamples them using grid downsampling. ResNet and FPN are responsible for extracting image features, while KPFCNN is responsible for extracting point cloud features. We use the points P^∈R^(N^×3) corresponding to the coarsest layer and their associated features F^^p∈R^(N^×C^). Wherein, given the downsampled image... And point cloud The goal of the coarse matching layer is to extract overlapping correspondences. However, inter-modal matching between 2D and 3D is not straightforward. Firstly, 2D and 3D features are learned from different domains, leading to significant inconsistencies between them. This problem is more severe in patch matching than in point matching because patch features are learned in a large context, exacerbating feature misalignment. Secondly, coarse matching relaxes the matching criterion from strict 3D distance to a more lenient local texture geometric similarity. This effectively reduces the matching difficulty but requires more global context. For this reason, the above embodiment designs a transformer-based feature refinement module to learn global context constraints and cross-modal associations.

[0100] Among them, the multi-scale sampling and matching strategy based on image spatial pyramids effectively alleviates the scale ambiguity problem caused by perspective projection and improves matching accuracy by segmenting the image into blocks of different scales and matching them with point cloud features. The coarse-to-fine matching process refines the matching process step by step, eliminating the need for keypoint detection between two modalities. It constructs a multi-scale block pyramid to further alleviate scale ambiguity in the block matching process, thereby improving matching accuracy.

[0101] The above embodiments address the shortcomings of existing 2D-3D correspondence visual localization methods in surveillance scenarios, including poor matching robustness, inability to effectively resolve scale ambiguity caused by perspective projection, the need for manual design of feature descriptors and matching strategies, and difficulty in adapting to complex scenes and large-scale data. To address these shortcomings, a multi-scale visual localization method based on Transformer is proposed. This method automatically learns effective features from images and point clouds through deep learning and Transformer networks, and refines these features using Transformer networks to improve their discriminative power and robustness, making them more resistant to the effects of lighting, occlusion, and other factors. To resolve the scale ambiguity caused by perspective projection, this invention designs a multi-scale sampling and matching strategy based on an image spatial pyramid. By segmenting the image into blocks of different scales and matching them with point cloud features, matching ambiguity can be effectively alleviated and learning difficulty reduced, thereby improving matching accuracy and robustness. Furthermore, the above embodiments employ a coarse-to-fine matching process. By progressively refining the matching process, erroneous matching relationships can be gradually eliminated, ultimately obtaining a more accurate 2D-3D correspondence, thus improving matching accuracy. Transformer networks possess excellent parallel computing capabilities, which can accelerate feature extraction and matching processes and improve computational efficiency. Simultaneously, multi-scale sampling and matching strategies can reduce learning difficulty and computational load, further enhancing computational efficiency.

[0102] Compared with existing 2D-3D correspondence visual localization methods, the above embodiments have the following significant advantages:

[0103] (1) Enhanced matching robustness: By using a Transformer network for feature refinement and multi-scale sampling, this invention can effectively extract deep features and contextual information from images and point clouds, thereby improving the robustness of features to factors such as illumination changes, occlusion, and background interference. This enables the invention to better adapt to more complex scenarios and large-scale data, such as in applications like autonomous driving and robot navigation, where it can more accurately identify and locate target objects.

[0104] (2) Higher matching accuracy: This invention, through a multi-scale sampling strategy and a coarse-to-fine matching process, can more comprehensively consider the correspondence between images and point clouds, and gradually eliminate erroneous matching relationships, thereby obtaining more accurate 2D-3D correspondences. In addition, the feature refinement capability of the Transformer network can extract richer feature information, further improving the matching accuracy.

[0105] (3) Simplified algorithm flow: This invention eliminates the need for manual design of feature descriptors and matching strategies. Instead, it utilizes deep learning methods to automatically learn features and matching relationships, simplifying the algorithm flow, reducing development costs, and improving the algorithm's scalability. This allows the invention to adapt to new scenarios and target objects more quickly.

[0106] The technical solution of this application, the aforementioned multi-scale visual positioning method, first acquires 2D image data and point cloud data within the monitoring range; then, it uses a ResNet network to extract features from the 2D image data, obtaining 2D image features; next, it uses a KPFCNN network to extract features from the point cloud data, obtaining point cloud features; finally, it uses a Transformer network to perform feature matching on the 2D image features and point cloud features, obtaining feature matching results, and determining the three-dimensional coordinates of the target object based on the feature matching results. This method utilizes deep learning methods to automatically learn effective features in images and point clouds, and refines these features through a Transformer network, improving the discriminative power and robustness of the features, making them more resistant to the influence of factors such as illumination and occlusion. It solves the problems of low matching accuracy, poor robustness, and inability to effectively resolve scale ambiguities caused by perspective projection in existing 2D-3D correspondence visual positioning methods.

[0107] This application also provides a multi-scale visual positioning device. It should be noted that the multi-scale visual positioning device of this application can be used to execute the multi-scale visual positioning method provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0108] The following describes the multi-scale visual positioning device provided in the embodiments of this application.

[0109] Figure 5 This is a schematic diagram of a multi-scale visual positioning device according to an embodiment of this application. Figure 5As shown, the device includes an acquisition unit 10, a first extraction unit 20, a second extraction unit 30, and a determination unit 40. The acquisition unit 10 is used to acquire 2D image data and point cloud data within the monitoring range. The first extraction unit 20 is used to extract features from the 2D image data using a ResNet network to obtain 2D image features. The second extraction unit 30 is used to extract features from the point cloud data using a KPFCNN network to obtain point cloud features. The determination unit 40 is used to perform feature matching on the 2D image features and the point cloud features using a Transformer network to obtain feature matching results, and determines the three-dimensional coordinates of the target object based on the feature matching results.

[0110] The multi-scale visual positioning device of this application includes an acquisition unit, a first extraction unit, a second extraction unit, and a determination unit. The acquisition unit acquires 2D image data and point cloud data within the monitoring range. The first extraction unit uses a ResNet network to extract features from the 2D image data, obtaining 2D image features. The second extraction unit uses a KPFCNN network to extract features from the point cloud data, obtaining point cloud features. The determination unit uses a Transformer network to perform feature matching on the 2D image features and point cloud features, obtaining feature matching results, and determining the three-dimensional coordinates of the target object based on the feature matching results. This device automatically learns effective features in images and point clouds using deep learning methods and refines features through a Transformer network, improving the discriminative power and robustness of the features, making them more resistant to the influence of factors such as illumination and occlusion. It solves the problems of low matching accuracy, poor robustness, and inability to effectively resolve scale ambiguity caused by perspective projection in existing 2D-3D correspondence visual positioning methods.

[0111] In some instances, the determining unit includes a first enhancement module, a first determining module, a second determining module, and a first matching module. The first enhancement module enhances the image features to obtain enhanced image features and enhances the point cloud features to obtain enhanced point cloud features. The first determining module determines an anchor feature matrix based on the enhanced image features, where the elements in the anchor feature matrix are the enhanced image features. The second determining module determines a memory feature matrix based on the enhanced point cloud features, where the elements in the memory feature matrix are the enhanced point cloud features. The first matching module uses a Transformer network and an attention mechanism to model the pairwise association between the elements in the anchor feature matrix and the elements in the memory feature matrix to obtain the feature matching result.

[0112] In some instances, the first enhancement module includes a first acquisition module, a second acquisition module, a second enhancement module, and a third enhancement module. The first acquisition module is used to acquire the first location information enhancement formula. in, These are 2D image features enhanced with location information. For the 2D image features, The first module represents the pixel coordinate matrix of the downsampled image, and φ(·) represents the Fourier embedding function. The second module is used to obtain the second location information enhancement formula. in, The point cloud features are augmented with location information. For the point cloud features, The first enhancement module is used to enhance the image features using the first location information enhancement formula to obtain the enhanced image features; the second enhancement module is used to enhance the point cloud features using the second location information enhancement formula to obtain the enhanced point cloud features.

[0113] In some instances, the first matching module includes a third acquisition module, a projection module, and a second matching module. The third acquisition module is used to acquire a first projection weight, a second projection weight, and a third projection weight. The first projection weight is the projection weight of the query condition, the second projection weight is the projection weight of the feature key, and the third projection weight is the projection weight of the feature value. The projection module is used to project the anchor feature matrix and the memory feature matrix using the first projection weight, the second projection weight, and the third projection weight to obtain the relevance between each element in the anchor feature matrix and each element in the memory feature matrix. The second matching module is used to input the relevance into a shallow MLP network to further project attention features to obtain the feature matching result.

[0114] In some instances, the projection module includes a fourth acquisition module, a fifth acquisition module, and a third determination module. The fourth acquisition module is used to obtain the query condition calculation formula Q = F. A W Q The formula for calculating characteristic keys is K = F. M W K The formula for calculating eigenvalues ​​is V = F. M W V Where Q is the query condition, K is the feature key, V is the feature value, and F is the feature value. A F is an element in the anchor point feature matrix. M For the elements in the memory feature matrix, W Q W is the first projection weight. K W is the second projection weight. V The third projection weight; the fifth acquisition module is used to acquire the attention feature formula. Among them, Attention(F A FM ) is F A and F M The relevance between the elements is given by: Q being the query condition, K being the feature key, V being the feature value, and d being the number of columns in the anchor feature matrix and the memory feature matrix. The third determining module is used to determine the relevance between each element in the anchor feature matrix and each element in the memory feature matrix according to the attention feature formula.

[0115] In some instances, the third acquisition module includes a first acquisition submodule, a second acquisition submodule, and a first determination submodule. The first acquisition submodule is used to acquire the initial first weight, the initial second weight, and the initial third weight; the second acquisition submodule is used to acquire the model's total loss formula L. all =L coarse +λL fine , where L all L represents the total loss of the model. coarse Let L be the first circumferential loss, λ be the balance factor, and L be the circumferential loss. fine The second circular loss is related to the initial first weight, the initial second weight, and the initial third weight. The first circular loss is a circular loss calculated based on the patch feature distance at a coarse level, where the coarse level is defined by the overlap ratio of positive and negative samples based on the patch feature distance. The second circular loss is a circular loss calculated based on the feature distance between pixels and points at a fine level, where the fine level is defined by the spatial distance between positive and negative samples based on the spatial distance between pixels and points. The determining submodule is used to determine the total model loss according to the formula for total model loss, and adjust the initial first weight, the initial second weight, and the initial third weight to minimize the total model loss. The initial first weight, the initial second weight, and the initial third weight corresponding to the minimum total model loss are determined as the first projection weight, the second projection weight, and the third projection weight.

[0116] In some instances, the apparatus further includes a third acquisition submodule and a second determination submodule, the third acquisition submodule being used to acquire the circumferential loss formula before acquiring the total model loss formula. Where, d i For anchor descriptors, L i For d i The corresponding circumferential loss value, one anchor descriptor corresponds to one circumferential loss value. It is the Euclidean distance between the i-th sample and the j-th sample. and These are the individual weights of positive and negative sample pairs, while and It is the ratio factor between positive and negative sample pairs, Δ pΔ n These are the expected distances between positive samples and the expected distances between negative samples; the second determining submodule is used to determine the sum of all the said circumferential loss values ​​as the circumferential loss.

[0117] The multi-scale visual positioning device includes a processor and a memory. The aforementioned acquisition units are all stored as program units in the memory, and the processor executes these program units to achieve the corresponding functions. All of the above modules are located in the same processor; alternatively, the modules may be located in different processors in any combination.

[0118] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and adjusting kernel parameters can address the problems of low matching accuracy, poor robustness, and inability to effectively resolve scale ambiguity caused by perspective projection in existing 2D-3D correspondence visual positioning methods.

[0119] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0120] This invention provides a computer-readable storage medium including a stored program, wherein the program, when running, controls the device where the computer-readable storage medium is located to execute the multi-scale visual positioning method.

[0121] This invention provides a processor for running a program, wherein the program executes the multi-scale visual localization method during runtime.

[0122] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements at least the steps of a multi-scale visual localization method. The device described herein may be a server, PC, PAD, mobile phone, etc.

[0123] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform a program that initializes a method with at least a multi-scale visual positioning method.

[0124] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0125] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0129] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0130] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0131] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0132] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0133] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0134] 1) The technical solution of this application, the aforementioned multi-scale visual positioning method, firstly acquires 2D image data and point cloud data within the monitoring range; then, it uses a ResNet network to extract features from the 2D image data to obtain 2D image features; next, it uses a KPFCNN network to extract features from the point cloud data to obtain point cloud features; finally, it uses a Transformer network to perform feature matching on the 2D image features and point cloud features to obtain feature matching results, and determines the three-dimensional coordinates of the target object based on the feature matching results. This method utilizes deep learning methods to automatically learn effective features in images and point clouds, and refines the features through a Transformer network, improving the discriminative power and robustness of the features, making them more resistant to the influence of factors such as illumination and occlusion. It solves the problems of low matching accuracy, poor robustness, and inability to effectively resolve scale ambiguity caused by perspective projection in existing 2D-3D correspondence visual positioning methods.

[0135] 2) The multi-scale visual positioning device of this application includes an acquisition unit, a first extraction unit, a second extraction unit, and a determination unit. The acquisition unit is used to acquire 2D image data and point cloud data within the monitoring range; the first extraction unit is used to extract features from the 2D image data using a ResNet network to obtain 2D image features; the second extraction unit is used to extract features from the point cloud data using a KPFCNN network to obtain point cloud features; the determination unit is used to perform feature matching on the 2D image features and point cloud features using a Transformer network to obtain feature matching results, and determines the three-dimensional coordinates of the target object based on the feature matching results. This device automatically learns effective features in images and point clouds using deep learning methods, and refines the features through a Transformer network to improve the discriminative power and robustness of the features, making them more resistant to the influence of factors such as illumination and occlusion. It solves the problems of low matching accuracy, poor robustness, and inability to effectively resolve scale ambiguity caused by perspective projection in existing 2D-3D correspondence visual positioning methods.

[0136] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multi-scale visual localization method, characterized in that, include: Acquire 2D image data and point cloud data within the monitoring range; The features of the 2D image data are extracted using a ResNet network to obtain the 2D image features; The KPFCNN network is used to extract features from the point cloud data to obtain point cloud features; The Transformer network is used to perform feature matching on the 2D image features and the point cloud features to obtain the feature matching results, and the three-dimensional coordinates of the target object are determined based on the feature matching results. The image features and the point cloud features are matched using a Transformer network to obtain the feature matching results, including: The image features are enhanced to obtain enhanced image features, and the point cloud features are enhanced to obtain enhanced point cloud features; Based on the enhanced image features, an anchor feature matrix is ​​determined, wherein the elements in the anchor feature matrix are the enhanced image features; Based on the enhanced point cloud features, a memory feature matrix is ​​determined, wherein the elements in the memory feature matrix are the enhanced point cloud features; The Transformer network is used to model the pairwise association between the elements in the anchor feature matrix and the elements in the memory feature matrix through an attention mechanism, thereby obtaining the feature matching result; The Transformer network, through an attention mechanism, models the pairwise associations between elements in the anchor feature matrix and elements in the memory feature matrix to obtain the feature matching results, including: Obtain a first projection weight, a second projection weight, and a third projection weight, wherein the first projection weight is the projection weight of the query condition, the second projection weight is the projection weight of the feature key, and the third projection weight is the projection weight of the feature value; The anchor feature matrix and the memory feature matrix are projected using the first projection weight, the second projection weight, and the third projection weight to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix; The relevance is input into a shallow MLP network to further project attention features, thereby obtaining the feature matching result.

2. The method according to claim 1, characterized in that, The image features are enhanced to obtain enhanced image features, and the point cloud features are enhanced to obtain enhanced point cloud features, including: Enhanced formula for obtaining first position information ,in, These are 2D image features enhanced with location information. For the 2D image features, For the pixel coordinate matrix of the downsampled image, For Fourier embedding functions; Formula for obtaining second position information enhancement ,in, The point cloud features are augmented with location information. For the point cloud features, These are the points corresponding to the coarsest layer in the KPFCNN network; The image features are enhanced using the first location information enhancement formula to obtain the enhanced image features; The point cloud features are enhanced using the second location information enhancement formula to obtain the enhanced point cloud features.

3. The method according to claim 1, characterized in that, The anchor feature matrix and the memory feature matrix are projected using the first projection weight, the second projection weight, and the third projection weight to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix, including: Get the calculation formula for the query conditions Characteristic key calculation formula Eigenvalue calculation formula Where Q is the query condition, K is the feature key, and V is the feature value. For the elements in the anchor point feature matrix, For the elements in the memory feature matrix, The first projection weight, The second projection weight, The third projection weight; Formula for obtaining attention features ,in, for and The relevance between them, Q is the query condition, K is the feature key, V is the feature value, and d is the number of columns in the anchor feature matrix and the memory feature matrix; The correlation between each element in the anchor feature matrix and each element in the memory feature matrix is ​​determined based on the attention feature formula.

4. The method according to claim 1, characterized in that, Obtaining the first projection weights, the second projection weights, and the third projection weights includes: Obtain the initial first weight, initial second weight, and initial third weight; Obtain the formula for total model loss ,in, The total loss of the model is given. For the first circumferential loss, As a balance factor, The second circular loss is related to the initial first weight, the initial second weight, and the initial third weight. The first circular loss is a circular loss calculated based on the patch feature distance at a coarse level, where the coarse level is defined by the overlap ratio of positive and negative samples based on the patch feature distance. The second circular loss is a circular loss calculated based on the feature distance between pixels and points at a fine level, where the fine level is defined by the spatial distance between positive and negative samples based on the pixel and point feature distance. According to the formula for total model loss, the total model loss is determined, and the initial first weight, the initial second weight, and the initial third weight are adjusted to minimize the total model loss. The initial first weight, the initial second weight, and the initial third weight corresponding to the minimum total model loss are then determined as the first projection weight, the second projection weight, and the third projection weight.

5. The method according to claim 4, characterized in that, Before obtaining the formula for the total loss of the model, the method further includes: Formula for obtaining circumferential loss ,in, For anchor descriptors, for The corresponding circumferential loss value, one anchor descriptor corresponds to one circumferential loss value. It is the Euclidean distance between the i-th sample and the j-th sample. and These are the individual weights of positive and negative sample pairs, while and It is the ratio factor between positive and negative sample pairs. These are the expected distances between positive samples and the expected distances between negative samples; The sum of all the circumferential loss values ​​is determined as the circumferential loss.

6. A multi-scale visual positioning device, characterized in that, include: The acquisition unit is used to acquire 2D image data and point cloud data within the monitoring range; The first extraction unit is used to extract features from the 2D image data using a ResNet network to obtain 2D image features; The second extraction unit is used to extract features from the point cloud data using a KPFCNN network to obtain point cloud features. The determining unit is used to perform feature matching on the 2D image features and the point cloud features using a Transformer network to obtain feature matching results, and to determine the three-dimensional coordinates of the target object based on the feature matching results; The determining unit includes a first enhancement module, a first determining module, a second determining module, and a first matching module. The first enhancement module is used to enhance the image features to obtain enhanced image features, and to enhance the point cloud features to obtain enhanced point cloud features. The first determining module is used to determine an anchor feature matrix based on the enhanced image features, wherein the elements in the anchor feature matrix are the enhanced image features. The second determining module is used to determine a memory feature matrix based on the enhanced point cloud features, wherein the elements in the memory feature matrix are the enhanced point cloud features. The first matching module is used to model the pairwise association between the elements in the anchor feature matrix and the elements in the memory feature matrix using a Transformer network and an attention mechanism to obtain the feature matching result. The first matching module includes a third acquisition module, a projection module, and a second matching module. The third acquisition module is used to acquire a first projection weight, a second projection weight, and a third projection weight. The first projection weight is the projection weight of the query condition, the second projection weight is the projection weight of the feature key, and the third projection weight is the projection weight of the feature value. The projection module is used to project the anchor feature matrix and the memory feature matrix using the first projection weight, the second projection weight, and the third projection weight to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix. The second matching module is used to input the correlation into a shallow MLP network to further project attention features to obtain the feature matching result.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the multi-scale visual positioning method according to any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising the multi-scale visual localization method according to any one of claims 1 to 5.

Images