Multi-scale visual positioning method and device, storage medium and electronic equipment

Abstract

The invention provides a multi-scale visual positioning method and device, a storage medium and electronic equipment. The method comprises the steps that 2D image data and point cloud data in a monitoring range are acquired; extracting features of the 2D image data by adopting a ResNet network to obtain 2D image features; extracting features of the point cloud data by adopting a KPFCNN network to obtain point cloud features; and carrying out feature matching on the 2D image features and the point cloud features by adopting a Transform network to obtain a feature matching result, and determining three-dimensional coordinates of the target object according to the feature matching result. According to the method, effective features in images and point clouds are automatically learned by using a deep learning method, and feature refinement is carried out through a Transform network, so that the distinction degree and robustness of the features are improved, and the method is more resistant to the influence of factors such as illumination and shielding.

Description

Technical Field

[0001] The present application relates to the field of visual positioning, and in particular, to a multi-scale visual positioning method, a multi-scale visual positioning device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Traditional 2D-3D correspondence visual positioning methods are mainly divided into two categories:

[0003] 1. The feature matching-based method extracts feature points from the image and point cloud and matches them to establish a 2D-3D correspondence. The PnP method of SIFT / ORB feature matching is a typical representative. This type of method mainly relies on manually designed feature descriptors. For example, SIFT and ORB algorithms respectively design unique feature descriptors to describe local features in the image. This design process is time-consuming and labor-intensive, and it is difficult to ensure the applicability of feature descriptors to different scenes and targets. In addition, it is necessary to design matching strategies, such as nearest neighbor matching or brute force matching, to match image feature points with point cloud feature points. This type of strategy is easily affected by factors such as noise and occlusion, resulting in matching errors, and is difficult to adapt to complex scenes and large-scale data.

[0004] 2. The model matching-based method establishes a 2D-3D correspondence by matching the 3D model with the point cloud. For example, the ICP algorithm is a typical representative. This type of method requires the establishment of a 3D model in advance, which limits its scope of application. For some targets for which a 3D model cannot be obtained, this method cannot be used. In addition, the model matching-based method is difficult to adapt to dynamic changes in the scene, such as changes in the target's posture, occlusion, etc., resulting in a decrease in matching accuracy.

[0005] Therefore, the traditional 2D-3D correspondence visual positioning method has low matching accuracy, poor robustness, and cannot effectively solve the scale ambiguity problem caused by perspective projection. Summary of the invention

[0006] The main purpose of the present application is to provide a multi-scale visual positioning method, a multi-scale visual positioning device, a computer-readable storage medium and an electronic device, so as to at least solve the problems of low matching accuracy, poor robustness and inability to effectively solve the scale ambiguity caused by perspective projection in the 2D-3D correspondence visual positioning method in the prior art.

[0007] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a multi-scale visual positioning method is provided, including: acquiring 2D image data and point cloud data within a monitoring range; using a ResNet network to extract features of the 2D image data to obtain 2D image features; using a KPFCNN network to extract features of the point cloud data to obtain point cloud features; using a Transformer network to perform feature matching on the 2D image features and the point cloud features to obtain feature matching results, and determining the three-dimensional coordinates of the target object based on the feature matching results.

[0008] Optionally, a Transformer network is used to perform feature matching on the image features and the point cloud features to obtain feature matching results, including: enhancing the image features to obtain enhanced image features, and enhancing the point cloud features to obtain enhanced point cloud features; determining an anchor feature matrix based on the enhanced image features, and the elements in the anchor feature matrix are the enhanced image features; determining a memory feature matrix based on the enhanced point cloud features, and the elements in the memory feature matrix are the enhanced point cloud features; and using a Transformer network to model the relationship between the elements in the anchor feature matrix and the elements in the memory feature matrix through an attention mechanism to obtain the feature matching result.

[0009] Optionally, the image feature is enhanced to obtain an enhanced image feature, and the point cloud feature is enhanced to obtain an enhanced point cloud feature, including: obtaining a first position information enhancement formula in, is the 2D image feature after position information enhancement, is the 2D image feature, is the pixel coordinate matrix of the downsampled image, φ(·) is the Fourier embedding function; the second position information enhancement formula is obtained in, is the point cloud feature after location information enhancement. is the point cloud feature, is the point corresponding to the coarsest layer in the KPFCNN network; the image feature is enhanced by using the first position information enhancement formula to obtain the enhanced image feature; the point cloud feature is enhanced by using the second position information enhancement formula to obtain the enhanced point cloud feature.

[0010] Optionally, a Transformer network is used to model the pairwise association between elements in the anchor feature matrix and elements in the memory feature matrix through an attention mechanism to obtain the feature matching result, including: obtaining a first projection weight, a second projection weight and a third projection weight, wherein the first projection weight is a projection weight of a query condition, the second projection weight is a projection weight of a feature key, and the third projection weight is a projection weight of a feature value; using the first projection weight, the second projection weight and the third projection weight to project the anchor feature matrix and the memory feature matrix to obtain a correlation between each element in the anchor feature matrix and each element in the memory feature matrix; inputting the correlation into a shallow MLP network to further project attention features to obtain the feature matching result.

[0011] Optionally, the first projection weight, the second projection weight, and the third projection weight are used to project the anchor feature matrix and the memory feature matrix to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix, including: obtaining the query condition calculation formula Q=F A W Q 、Characteristic key calculation formula K=F M W K And the eigenvalue calculation formula V = F M W V , where Q is the query condition, K is the feature key, V is the feature value, and F A is the element in the anchor feature matrix, F M is the element in the memory feature matrix, W Q is the first projection weight, W K is the second projection weight, W V is the third projection weight; obtain the attention feature formula Among them, Attention(F A , F M ) is F A and F M where Q is the query condition, K is the feature key, V is the feature value, and d is the number of columns of the anchor feature matrix and the memory feature matrix; and the correlation between each element in the anchor feature matrix and each element in the memory feature matrix is ​​determined according to the attention feature formula.

[0012] Optionally, obtaining the first projection weight, the second projection weight and the third projection weight includes: obtaining an initial first weight, an initial second weight and an initial third weight; obtaining a model total loss formula L all =L coarse +λL fine , where Lall is the total loss of the model, L coarse is the first circle loss, λ is the balance factor, L fine is the second circular loss, the total model loss is related to the initial first weight, the initial second weight and the initial third weight, the first circular loss is the circular loss calculated based on the patch feature distance at a coarse level, the coarse level is the level defined by the positive and negative samples based on the overlap ratio of the block, the second circular loss is the circular loss calculated based on the feature distance of pixels and points at a fine level, the fine level is the positive and negative samples defined based on the spatial distance of pixels and points; according to the model total loss formula, the model total loss is determined, and the initial first weight, the initial second weight and the initial third weight are adjusted to minimize the model total loss, and the initial first weight, the initial second weight and the initial third weight corresponding to the minimum model total loss are determined as the first projection weight, the second projection weight and the third projection weight.

[0013] Optionally, before obtaining the total loss formula of the model, the method further includes: obtaining a circumferential loss formula Among them, d i is the anchor descriptor, L i is d i The corresponding circumferential loss value, one anchor descriptor corresponds to one circumferential loss value, is the Euclidean distance between the i-th sample and the j-th sample, and are the individual weights of positive and negative pairs, and and is the ratio factor between positive and negative sample pairs, Δ p , Δ n is the expected distance between positive samples and the expected distance between negative samples; and the sum of all the circumferential loss values ​​is determined as the circumferential loss.

[0014] According to another aspect of the present application, a multi-scale visual positioning device is provided, including: an acquisition unit, used to acquire 2D image data and point cloud data within a monitoring range; a first extraction unit, used to extract features of the 2D image data using a ResNet network to obtain 2D image features; a second extraction unit, used to extract features of the point cloud data using a KPFCNN network to obtain point cloud features; a determination unit, used to perform feature matching on the 2D image features and the point cloud features using a Transformer network to obtain feature matching results, and determine the three-dimensional coordinates of the target object according to the feature matching results.

[0015] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the multi-scale visual positioning methods.

[0016] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the multi-scale visual positioning methods.

[0017] Applying the technical solution of the present application, the above-mentioned multi-scale visual positioning method first obtains 2D image data and point cloud data within the monitoring range; then uses the ResNet network to extract the features of the 2D image data to obtain 2D image features; then uses the KPFCNN network to extract the features of the point cloud data to obtain point cloud features; finally, uses the Transformer network to perform feature matching on the 2D image features and point cloud features to obtain feature matching results, and determines the three-dimensional coordinates of the target object based on the feature matching results. This method uses deep learning methods to automatically learn effective features in images and point clouds, and refines features through the Transformer network to improve the distinguishability and robustness of features, making them more resistant to the influence of factors such as illumination and occlusion. It solves the problems of low matching accuracy, poor robustness, and inability to effectively solve the scale ambiguity caused by perspective projection in the 2D-3D correspondence visual positioning method in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 A hardware structure block diagram of a mobile terminal for executing a multi-scale visual positioning method provided in an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram of a multi-scale visual positioning method provided according to an embodiment of the present application is shown;

[0021] Figure 3 A network architecture diagram of a Transformer network provided according to an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of a multi-scale visual positioning method provided according to an embodiment of the present application is shown;

[0023] Figure 5 A structural block diagram of a multi-scale visual positioning device provided according to an embodiment of the present application is shown.

[0024] The above drawings include the following reference numerals:

[0025] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0030] 2D-3D correspondence: refers to the mapping relationship between pixel points in the image and corresponding points in the three-dimensional point cloud.

[0031] Transformer: A deep learning model that learns internal correlations in data through a self-attention mechanism.

[0032] ResNet: A deep convolutional neural network that solves the gradient vanishing problem in deep network training through residual connections.

[0033] FPN: A feature pyramid network that fuses feature maps at different levels to obtain feature maps of different scales.

[0034] KPFCNN: A point cloud learning network based on KPConv, a convolutional operation for point clouds.

[0035] Multi-scale sampling: Split the image into blocks of different scales to alleviate matching ambiguity.

[0036] Block pyramid: Each image block is divided into different levels to extract finer-grained features.

[0037] As introduced in the background technology, in recent years, deep learning methods have made significant progress in image recognition and 3D point cloud processing, providing new ideas for 2D-3D correspondence visual positioning. For example, feature extraction methods based on deep learning can automatically learn features in images and point clouds without manually designing feature descriptors; feature matching methods based on deep learning can use neural networks to learn the similarities between features and improve matching robustness; model matching methods based on deep learning can learn the parameters of 3D models without pre-building 3D models.

[0038] However, the existing 2D-3D correspondence visual positioning methods have certain technical defects, which limit their application in complex scenes and large-scale data.

[0039] Manually designed feature descriptors and matching strategies are difficult to adapt to complex scenarios and large-scale data.

[0040] Traditional feature descriptors, such as SIFT and ORB, need to be manually designed, which is a time-consuming and labor-intensive process that is difficult to ensure applicability. Different scenes and targets may require different feature descriptors, for example, SIFT is more suitable for scenes with rich textures, while ORB is more suitable for scenes with less textures. This design process not only requires a lot of time and expertise, but also makes it difficult to ensure the applicability of feature descriptors to different scenes and targets.

[0041] Traditional matching strategies, such as nearest neighbor matching or brute force matching, are easily affected by factors such as noise and occlusion, resulting in matching errors. For example, in complex scenarios such as lighting changes, occlusion, and background interference, the difficulty of extracting and matching feature points will be greatly increased, further reducing the robustness of matching. In addition, these matching strategies are difficult to adapt to the processing of large-scale data, which limits their efficiency in practical applications. Moreover, the feature extraction and matching process that requires manual intervention in traditional methods is difficult to adapt to the processing of large-scale data. For example, in application scenarios such as autonomous driving and robot navigation, a large amount of image and point cloud data needs to be processed, and the efficiency of traditional methods is difficult to meet actual needs.

[0042] At the same time, changes in illumination can cause changes in the brightness and contrast of feature points in images and point clouds, affecting the accuracy of feature point extraction and matching. For example, in direct sunlight or backlight, the characteristics of feature points may change significantly, resulting in matching failures; and occlusion can cause some feature points to be undetectable, affecting matching accuracy. For example, in scenes such as buildings and trees, occlusion is common, resulting in a decrease in the matching robustness of traditional methods; background interference can cause feature point extraction and matching to be mismatched, affecting positioning accuracy. For example, in scenes with complex backgrounds, such as city streets, the texture and color in the background may be similar to the target object, resulting in mismatches.

[0043] In addition, existing technologies cannot effectively solve the scale ambiguity problem caused by perspective projection. Perspective projection causes scale changes in objects in the image, that is, objects that are closer appear larger than objects that are farther away appear smaller. Traditional 2D-3D correspondence visual positioning methods are difficult to effectively solve this problem, resulting in reduced matching accuracy. Scale ambiguity can lead to matching errors, such as mismatching distant objects to nearby objects, or mismatching nearby objects to distant objects.

[0044] Existing 2D-3D correspondence visual positioning methods have technical defects such as manually designed feature descriptors and matching strategies, poor matching robustness, and inability to effectively solve the scale ambiguity problem caused by perspective projection. They are difficult to adapt to complex scenes and large-scale data applications.

[0045] In order to solve the problems of low matching accuracy, poor robustness and inability to effectively solve the scale ambiguity caused by perspective projection in the 2D-3D correspondence visual positioning method in the prior art, the embodiments of the present application provide a multi-scale visual positioning method, a multi-scale visual positioning device, a computer-readable storage medium and an electronic device.

[0046] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0047] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal of a multi-scale visual positioning method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0048] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the multi-scale visual positioning method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific network example may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] In this embodiment, a multi-scale visual positioning method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] Figure 2 is a flow chart of a multi-scale visual positioning method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0051] Step S201, acquiring 2D image data and point cloud data within the monitoring range;

[0052] Specifically, in the data preparation stage, 2D images and prior point cloud data of the monitoring scene are first collected. Image data can be captured by a monitoring camera and preprocessed, such as cropping unnecessary parts and adjusting the image size. Point cloud data can be scanned by devices such as lidar and preprocessed, such as removing noise and downsampling. In order to improve the generalization ability of the model, the image and point cloud data are annotated, including the location and category of the monitored object.

[0053] Step S202, using a ResNet network to extract features of the 2D image data to obtain 2D image features;

[0054] Specifically, given a pair of images and point clouds, two modality-specific encoder-decoder backbone networks are used for hierarchical feature extraction. For images, the present invention uses ResNet and Feature Pyramid Networks (FPN) to generate multi-scale image features. Given Q∈R^(H×W×2) as the original image pixel coordinate matrix, Q^∈R^(H×W×2) as the downsampled image pixel coordinate matrix, the downsampled 2D features of the minimum resolution are matched with the original resolution at coarse and fine levels.

[0055] Step S203, using a KPFCNN network to extract features of the point cloud data to obtain point cloud features;

[0056] Specifically, for point clouds, we use KPFCNN to learn 3D features. Unlike images with fixed resolution, point clouds usually have inconsistent sizes, and KPFCNN dynamically downsamples them through grid downsampling. ResNet and FPN are responsible for extracting image features, and KPFCNN is responsible for extracting point cloud features. We use the points P ∈ R (N × 3) corresponding to the coarsest layer and their associated features F ∈ R (N × C).

[0057] In order to obtain richer feature information, the above steps use ResNet network and KPFCNN network to extract features of images and point clouds respectively. ResNet network can effectively extract deep features in images, and construct feature pyramids through FPN structure to obtain feature maps of different scales. KPFCNN network can effectively extract local features in point clouds, and perform feature extraction and propagation through KPConv convolution operation, so as to obtain richer point cloud feature information.

[0058] Step S204: Use a Transformer network to perform feature matching on the 2D image features and the point cloud features to obtain feature matching results, and determine the three-dimensional coordinates of the target object according to the feature matching results.

[0059] Specifically, these features are input into the Transformer network for feature refinement to obtain richer feature information. The multi-scale feature maps extracted by ResNet, FPN and KPFCNN are used to solve the scale ambiguity problem caused by perspective projection. The Transformer network further improves the matching accuracy by learning the correlation between different scales.

[0060] In the real-time positioning stage, the above embodiment inputs the 2D image of the real-time monitoring scene and the prior point cloud into the trained model. The model automatically extracts features from the image and point cloud, and performs feature refinement and matching, and finally obtains the correspondence between pixels and point clouds. The image detection and segmentation technology is used to segment the person in the image to obtain the area where the person is located. The matching framework of the present invention is used to calculate the correspondence between each pixel point in the segmented area and the point cloud point, and each pixel point in the image segmented area is mapped to the point cloud, thereby obtaining the position coordinates of the person in 3D space.

[0061] In the image branch, in addition to the ResNet network, other deep convolutional neural networks such as VGG, Inception, DenseNet, etc. can be used to extract image features. In the point cloud branch, in addition to KPFCNN, point cloud processing networks such as PointNet, PointNet++, DeepGCN, etc. can be used to extract point cloud features. For Transformer feature refinement, in addition to using the standard Transformer structure, you can also consider using a multi-head self-attention mechanism, a Non-local neural network, or other models based on the self-attention mechanism for feature fusion and refinement. These alternatives can all achieve the purpose of the present invention, namely, to improve the accuracy and robustness of 2D-3D correspondence visual positioning.

[0062] The above steps extract the pixel-point correspondence between the image and the point cloud by gradually refining the matching process, without the need for key point detection between the two modalities, and construct a multi-scale block pyramid to alleviate the scale ambiguity in the block matching process. The coarse-to-fine matching process can gradually eliminate incorrect matching relationships and ultimately obtain a more accurate 2D-3D correspondence relationship.

[0063] The Transformer network is used to perform feature matching on the image features and the point cloud features to obtain feature matching results, including:

[0064] Step S301, enhancing the image features to obtain enhanced image features, and enhancing the point cloud features to obtain enhanced point cloud features;

[0065] The step of enhancing the image features to obtain enhanced image features and enhancing the point cloud features to obtain enhanced point cloud features comprises the following steps:

[0066] Step S3011, obtaining the first position information enhancement formula in, is the 2D image feature after position information enhancement, is the 2D image feature, is the pixel coordinate matrix of the downsampled image, φ(·) is the Fourier embedding function;

[0067] Step S3012, obtaining the second position information enhancement formula in, is the point cloud feature after location information enhancement. is the point cloud feature, is the point corresponding to the coarsest layer in the KPFCNN network;

[0068] Step S3013, using the first position information enhancement formula to enhance the image feature to obtain the enhanced image feature;

[0069] Step S3014: Use the second position information enhancement formula to enhance the point cloud feature to obtain the enhanced point cloud feature.

[0070] Specifically, Where L is the length of the embedding. To simplify the calculation, the first two spatial dimensions (height and width) of the 2D feature matrix are flattened into one dimension. For future calculations. It is the 2D image features and 3D point cloud features after location information enhancement.

[0071] Among them, enhancing image features and point cloud features can obtain more accurate image features and point cloud features.

[0072] Step S302, determining an anchor feature matrix according to the enhanced image features, wherein the elements in the anchor feature matrix are the enhanced image features;

[0073] Step S303, determining a memory feature matrix according to the enhanced point cloud features, wherein the elements in the memory feature matrix are the enhanced point cloud features;

[0074] Step S304, using the Transformer network through the attention mechanism, modeling the association between the elements in the anchor feature matrix and the elements in the memory feature matrix, and obtaining the feature matching result.

[0075] Specifically, transformers are used to further refine features in two ways. Given an anchor feature matrix F A ∈R |A|×d and memory feature matrix F M ∈R |M|×d , transformer models the pairwise associations between them through the attention mechanism to generate more discriminative features.

[0076] Use the Transformer network for feature refinement. The Transformer network can effectively capture long-distance dependencies and learn the correlation between different modalities through self-attention and cross-attention mechanisms, thereby extracting richer feature information. This can further improve the distinguishability and robustness of features.

[0077] The Transformer network is used for feature refinement, and the contextual information and cross-modal correlation within the image and point cloud are learned through the self-attention mechanism and cross-attention mechanism to improve the distinguishability and robustness of the features, thereby improving the matching accuracy and robustness.

[0078] The Transformer network is used to model the relationship between the elements in the anchor feature matrix and the elements in the memory feature matrix through the attention mechanism to obtain the feature matching result, including the following steps:

[0079] Step S401, obtaining a first projection weight, a second projection weight, and a third projection weight, wherein the first projection weight is a projection weight of a query condition, the second projection weight is a projection weight of a feature key, and the third projection weight is a projection weight of a feature value;

[0080] The step of obtaining the first projection weight, the second projection weight and the third projection weight comprises the following steps:

[0081] Step S4011, obtaining an initial first weight, an initial second weight, and an initial third weight;

[0082] Step S4012, obtain the total loss formula of the model L all =L coarse +λL fine , where L all is the total loss of the model, L coarse is the first circle loss, λ is the balance factor, L fineis a second circular loss, the total model loss is related to the initial first weight, the initial second weight and the initial third weight, the first circular loss is a circular loss calculated based on the patch feature distance at a coarse level, the coarse level is a level defined by the overlap ratio of the positive and negative samples based on the block, the second circular loss is a circular loss calculated based on the feature distance of pixels and points at a fine level, the fine level is a positive and negative sample based on the spatial distance of pixels and points defined;

[0083] Step S4013, according to the model total loss formula, determine the model total loss, and adjust the initial first weight, the initial second weight and the initial third weight to minimize the model total loss, and determine the initial first weight, the initial second weight and the initial third weight corresponding to the minimum model total loss as the first projection weight, the second projection weight and the third projection weight.

[0084] Specifically, compared with contrastive loss and triplet loss, circle loss has a circular decision boundary, which is conducive to convergence. Given an anchor descriptor d i , whose positive and negative pair descriptors are expressed as and

[0085] Before obtaining the total loss formula of the model, the method further includes the following steps:

[0086] Step S501, obtain the circumferential loss formula Among them, d i is the anchor descriptor, L i is d i Corresponding to the circumferential loss value, one anchor descriptor corresponds to one circumferential loss value, is the Euclidean distance between the i-th sample and the j-th sample, and are the individual weights of positive and negative pairs, and and is the ratio factor between positive and negative sample pairs, Δ p , Δ n is the expected distance between positive samples and the expected distance between negative samples;

[0087] Step S502: determining the sum of all the circumferential loss values ​​as the circumferential loss.

[0088] Specifically, the model is trained in a metric learning manner. The model architecture is as follows Figure 3As shown in Figure 2. At the coarse level, a scaled circle loss is used to supervise the patch features. At the fine level, another standard circle loss is used to supervise the dense pixel and point features. The total loss is then calculated.

[0089] Step S402, projecting the anchor feature matrix and the memory feature matrix using the first projection weight, the second projection weight, and the third projection weight to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix;

[0090] The method of projecting the anchor feature matrix and the memory feature matrix using the first projection weight, the second projection weight, and the third projection weight to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix comprises the following steps:

[0091] Step S4021, obtain the query condition calculation formula Q=F A W Q 、Characteristic key calculation formula K=F M W K And the eigenvalue calculation formula V = F M W V , where Q is the query condition, K is the feature key, V is the feature value, and F A is the element in the anchor feature matrix, F M is the element in the memory feature matrix, W Q is the first projection weight, W K is the second projection weight, W V is the third projection weight;

[0092] Step S4022, obtain the attention feature formula Among them, Attention(F A , F M ) is F A and F M The correlation between them, Q is the query condition, K is the feature key, V is the feature value, d is...;

[0093] Step S4023, determining the correlation between each element in the anchor feature matrix and each element in the memory feature matrix according to the attention feature formula.

[0094] Specifically, in the model training phase, a Transformer-based multi-scale feature matching model is trained. Q , W K , W V ∈R d×dis the projection weight of Query, Key, and Value. The model includes an image branch and a point cloud branch, which are used to extract the features of the image and the point cloud, respectively. In order to improve the distinguishability and robustness of the features, the present invention uses the Transformer network to refine the features, and learns the contextual information and cross-modal correlation inside the image and point cloud through the self-attention mechanism and the cross-attention mechanism. In order to solve the scale ambiguity problem caused by perspective projection, the present invention designs a multi-scale sampling and matching strategy based on the image space pyramid. By dividing the image into blocks of different scales and matching them with the point cloud features, the matching ambiguity can be effectively alleviated and the learning difficulty can be reduced, thereby improving the matching accuracy and robustness. In addition, a coarse-to-fine matching process is adopted. By gradually refining the matching process, there is no need to perform key point detection between the two modalities, and a multi-scale block pyramid is constructed to further alleviate the scale ambiguity in the block matching process, thereby improving the matching accuracy. In order to improve the computational efficiency, the Adam optimization algorithm is used to train the model, and the model performance is evaluated and tuned using methods such as cross-validation.

[0095] Step S403: input the correlation into a shallow MLP network to further project the attention feature to obtain the feature matching result.

[0096] Specifically, a shallow MLP is used to further project the attention features as the final output features. The best match is found by comparing image patches and 3D patches at different scales. A mutual top-k selection method is used to extract rough patch correspondences. These correspondences represent preliminary matches between image patches and 3D patches.

[0097] For each coarse correspondence, local pixel and point features at fine level are collected from the image and point cloud. The collected 2D and 3D features are normalized to unit length vectors for easy comparison.

[0098] The above steps then match pixels and points using a mutual top-k selection method to generate local dense correspondences. This approach is more efficient than using a dedicated matching layer (such as Sinkhorn), especially when the 2D patches are large. All local correspondences are aggregated to form the final set of dense pixel-point correspondences.

[0099] In some instances, such as Figure 4As shown, given a pair of images and point clouds, two modality-specific encoder-decoder backbone networks are used for hierarchical feature extraction. For images, the present invention uses ResNet and Feature Pyramid Networks (FPN) to generate multi-scale image features. Given Q∈R^(H×W×2) as the original image pixel coordinate matrix, Q^∈R^(H×W×2) as the downsampled image pixel coordinate matrix, the downsampled 2D features of the minimum resolution are matched with the original resolution at coarse and fine levels. For point clouds, we use KPFCNN to learn three-dimensional features. Unlike images with fixed resolution, point clouds usually have inconsistent sizes, and KPFCNN dynamically downsamples them through grid downsampling. ResNet and FPN are responsible for extracting image features, and KPFCNN is responsible for extracting point cloud features. We use the points P^∈R^(N^×3) corresponding to the coarsest layer and its related features F^^p∈R^(N^×C^). Among them, given the downsampled image and point cloud The goal of the coarse matching layer is to extract mutually overlapping correspondences. However, inter-modal matching between 2D and 3D is not simple. On the one hand, 2D and 3D features are learned from different fields, resulting in serious inconsistencies between them. This problem is more serious in patch matching than in point matching because patch features are learned in a large environment, which exacerbates the misalignment of features. Secondly, coarse-level matching relaxes the matching criteria from strict 3D distance to a more relaxed local texture geometric similarity. This effectively reduces the matching difficulty, but requires more global context. For this reason, the above embodiment designs a transformer-based feature refinement module to learn global context constraints and cross-modal associations.

[0100] Among them, the multi-scale sampling and matching strategy based on the image space pyramid effectively alleviates the scale ambiguity problem caused by perspective projection and improves the matching accuracy by dividing the image into blocks of different scales and matching them with point cloud features. The coarse-to-fine matching process gradually refines the matching process without the need for key point detection between the two modalities, and constructs a multi-scale block pyramid, which further alleviates the scale ambiguity in the block matching process and thus improves the matching accuracy.

[0101] The above embodiment aims at the technical defects of the existing 2D-3D correspondence visual positioning method in the monitoring scene, such as poor matching robustness, inability to effectively solve the scale ambiguity problem caused by perspective projection, the need to manually design feature descriptors and matching strategies, and difficulty in adapting to complex scenes and large-scale data. A multi-scale visual positioning method based on Transformer is proposed. The method automatically learns effective features in images and point clouds through deep learning and Transformer networks, and refines features through Transformer networks to improve the distinguishability and robustness of features, making them more resistant to the influence of factors such as illumination and occlusion. In order to solve the scale ambiguity problem caused by perspective projection, the present invention designs a multi-scale sampling and matching strategy based on image space pyramid. By dividing the image into blocks of different scales and matching them with point cloud features, matching ambiguity can be effectively alleviated and learning difficulty can be reduced, thereby improving matching accuracy and robustness. In addition, the above embodiment adopts a coarse-to-fine matching process. By gradually refining the matching process, erroneous matching relationships can be gradually eliminated, and finally a more accurate 2D-3D correspondence relationship is obtained, thereby improving matching accuracy. The Transformer network has good parallel computing capabilities, which can accelerate the feature extraction and matching process and improve computing efficiency. At the same time, multi-scale sampling and matching strategies can reduce the difficulty of learning, reduce the amount of calculation, and further improve computing efficiency.

[0102] Compared with the existing 2D-3D correspondence visual positioning method, the above embodiment has the following significant advantages:

[0103] (1) Stronger matching robustness: By using the Transformer network for feature refinement and multi-scale sampling strategy, the present invention can effectively extract deep features and contextual information in images and point clouds, thereby improving the robustness of features to factors such as lighting changes, occlusion, and background interference. This enables the present invention to better adapt to more complex scenarios and large-scale data, such as in application scenarios such as autonomous driving and robot navigation, and to more accurately identify and locate target objects.

[0104] (2) Higher matching accuracy: Through a multi-scale sampling strategy and a coarse-to-fine matching process, the present invention can more comprehensively consider the correspondence between the image and the point cloud, and gradually eliminate erroneous matching relationships, thereby obtaining a more accurate 2D-3D correspondence. In addition, the feature refinement capability of the Transformer network can extract richer feature information, further improving matching accuracy.

[0105] (3) Simplified algorithm flow: The present invention does not require manual design of feature descriptors and matching strategies, but uses deep learning methods to automatically learn features and matching relationships, which simplifies the algorithm flow, reduces development costs, and improves the scalability of the algorithm. This enables the present invention to adapt to new scenarios and target objects more quickly.

[0106] The technical solution of the present application, the above-mentioned multi-scale visual positioning method, first obtains 2D image data and point cloud data within the monitoring range; then uses the ResNet network to extract the features of the 2D image data to obtain 2D image features; then uses the KPFCNN network to extract the features of the point cloud data to obtain point cloud features; finally, uses the Transformer network to perform feature matching on the 2D image features and point cloud features to obtain feature matching results, and determines the three-dimensional coordinates of the target object based on the feature matching results. This method uses deep learning methods to automatically learn effective features in images and point clouds, and refines features through the Transformer network to improve the distinguishability and robustness of features, making them more resistant to the influence of factors such as illumination and occlusion. It solves the problems of low matching accuracy, poor robustness, and inability to effectively solve the scale ambiguity caused by perspective projection in the 2D-3D correspondence visual positioning method in the prior art.

[0107] The embodiment of the present application also provides a multi-scale visual positioning device. It should be noted that the multi-scale visual positioning device of the embodiment of the present application can be used to execute the multi-scale visual positioning method provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions have been omitted. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0108] The following is an introduction to the multi-scale visual positioning device provided in the embodiments of the present application.

[0109] Figure 5 Schematic diagram of a multi-scale visual positioning device according to an embodiment of the present application. Figure 5As shown, the device includes an acquisition unit 10, a first extraction unit 20, a second extraction unit 30 and a determination unit 40. The acquisition unit 10 is used to acquire 2D image data and point cloud data within the monitoring range; the first extraction unit 20 is used to extract features of the 2D image data using a ResNet network to obtain 2D image features; the second extraction unit 30 is used to extract features of the point cloud data using a KPFCNN network to obtain point cloud features; the determination unit 40 is used to perform feature matching on the 2D image features and the point cloud features using a Transformer network to obtain feature matching results, and determine the three-dimensional coordinates of the target object according to the feature matching results.

[0110] The above-mentioned multi-scale visual positioning device of the present application includes an acquisition unit, a first extraction unit, a second extraction unit and a determination unit. The acquisition unit is used to acquire 2D image data and point cloud data within the monitoring range; the first extraction unit is used to extract the features of the 2D image data using the ResNet network to obtain 2D image features; the second extraction unit is used to extract the features of the point cloud data using the KPFCNN network to obtain point cloud features; the determination unit is used to perform feature matching on the 2D image features and the point cloud features using the Transformer network to obtain feature matching results, and determine the three-dimensional coordinates of the target object according to the feature matching results. The device uses a deep learning method to automatically learn effective features in images and point clouds, and refines features through a Transformer network to improve the distinguishability and robustness of features, making them more resistant to the influence of factors such as illumination and occlusion. It solves the problems of low matching accuracy, poor robustness, and inability to effectively solve the scale ambiguity caused by perspective projection in the 2D-3D correspondence visual positioning method in the prior art.

[0111] In some instances, the determination unit includes a first enhancement module, a first determination module, a second determination module and a first matching module, wherein the first enhancement module is used to enhance the image features to obtain enhanced image features, and enhance the point cloud features to obtain enhanced point cloud features; the first determination module is used to determine an anchor feature matrix according to the enhanced image features, and the elements in the anchor feature matrix are the enhanced image features; the second determination module is used to determine a memory feature matrix according to the enhanced point cloud features, and the elements in the memory feature matrix are the enhanced point cloud features; the first matching module is used to use a Transformer network to model the relationship between the elements in the anchor feature matrix and the elements in the memory feature matrix through an attention mechanism to obtain the feature matching result.

[0112] In some examples, the first enhancement module includes a first acquisition module, a second acquisition module, a second enhancement module and a third enhancement module, and the first acquisition module is used to acquire the first position information enhancement formula in, is the 2D image feature after position information enhancement, is the 2D image feature, is the downsampled image pixel coordinate matrix, φ(·) is the Fourier embedding function; the second acquisition module is used to obtain the second position information enhancement formula in, is the point cloud feature after location information enhancement. is the point cloud feature, is the point corresponding to the coarsest layer in the KPFCNN network; the second enhancement module is used to enhance the image features by using the first position information enhancement formula to obtain the enhanced image features; the third enhancement module is used to enhance the point cloud features by using the second position information enhancement formula to obtain the enhanced point cloud features.

[0113] In some instances, the first matching module includes a third acquisition module, a projection module and a second matching module, the third acquisition module is used to acquire a first projection weight, a second projection weight and a third projection weight, the first projection weight is a projection weight of a query condition, the second projection weight is a projection weight of a feature key, and the third projection weight is a projection weight of a feature value; the projection module is used to use the first projection weight, the second projection weight and the third projection weight to project the anchor feature matrix and the memory feature matrix to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix; the second matching module is used to input the correlation into a shallow MLP network to further project the attention feature to obtain the feature matching result.

[0114] In some examples, the projection module includes a fourth acquisition module, a fifth acquisition module and a third determination module, and the fourth acquisition module is used to obtain the query condition calculation formula Q=F A W Q 、Characteristic key calculation formula K=F M W K And the eigenvalue calculation formula V = F M W V , where Q is the query condition, K is the feature key, V is the feature value, and F A is the element in the anchor feature matrix, F M is the element in the memory feature matrix, W Q is the first projection weight, W K is the second projection weight, W V is the third projection weight; the fifth acquisition module is used to obtain the attention feature formula Among them, Attention(F A , FM ) is F A and F M The third determination module is used to determine the correlation between each element in the anchor feature matrix and each element in the memory feature matrix according to the attention feature formula.

[0115] In some examples, the third acquisition module includes a first acquisition submodule, a second acquisition submodule and a first determination submodule, the first acquisition submodule is used to obtain the initial first weight, the initial second weight and the initial third weight; the second acquisition submodule is used to obtain the total loss formula of the model L all =L coarse +λL fine , where L all is the total loss of the model, L coarse is the first circle loss, λ is the balance factor, L fine The second circular loss is the circular loss calculated based on the feature distance of patches at a coarse level, and the coarse level is the level defined by the overlap ratio of positive and negative samples based on the blocks; the second circular loss is the circular loss calculated based on the feature distance of pixels and points at a fine level, and the fine level is the level defined by the spatial distance of positive and negative samples based on pixels and points; the determination submodule is used to determine the total model loss according to the total model loss formula, and adjust the initial first weight, the initial second weight and the initial third weight to minimize the total model loss, and determine the initial first weight, the initial second weight and the initial third weight corresponding to the minimum total model loss as the first projection weight, the second projection weight and the third projection weight.

[0116] In some examples, the device further includes a third acquisition submodule and a second determination submodule, wherein the third acquisition submodule is used to obtain a circumferential loss formula before obtaining the model total loss formula. Among them, d i is the anchor descriptor, L i is d i The corresponding circumferential loss value, one anchor descriptor corresponds to one circumferential loss value, is the Euclidean distance between the i-th sample and the j-th sample, and are the individual weights of positive and negative pairs, and and is the ratio factor between positive and negative sample pairs, Δ p, Δ n is the expected distance between positive samples and the expected distance between negative samples; the second determination submodule is used to determine the sum of all the circumferential loss values ​​as the circumferential loss.

[0117] The multi-scale visual positioning device includes a processor and a memory, and the acquisition unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0118] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problems of low matching accuracy, poor robustness, and inability to effectively solve the scale ambiguity caused by perspective projection in the 2D-3D correspondence visual positioning method in the prior art.

[0119] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0120] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the multi-scale visual positioning method.

[0121] An embodiment of the present invention provides a processor, which is used to run a program, wherein the multi-scale visual positioning method is executed when the program is run.

[0122] The embodiment of the present invention provides a device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements at least the steps of the multi-scale visual positioning method when executing the program. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0123] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the steps of a multi-scale visual localization method.

[0124] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0125] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0126] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0129] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0130] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0131] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0132] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0133] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0134] 1) The technical solution of the present application, the above-mentioned multi-scale visual positioning method, first obtains the 2D image data and point cloud data within the monitoring range; then uses the ResNet network to extract the features of the 2D image data to obtain the 2D image features; then uses the KPFCNN network to extract the features of the point cloud data to obtain the point cloud features; finally, uses the Transformer network to perform feature matching on the 2D image features and the point cloud features to obtain the feature matching results, and determines the three-dimensional coordinates of the target object based on the feature matching results. This method uses a deep learning method to automatically learn effective features in images and point clouds, and refines features through a Transformer network to improve the distinguishability and robustness of features, making them more resistant to the influence of factors such as illumination and occlusion. It solves the problems of low matching accuracy, poor robustness, and inability to effectively solve the scale ambiguity caused by perspective projection in the 2D-3D correspondence visual positioning method in the prior art.

[0135] 2) The multi-scale visual positioning device of the present application includes an acquisition unit, a first extraction unit, a second extraction unit and a determination unit. The acquisition unit is used to acquire 2D image data and point cloud data within the monitoring range; the first extraction unit is used to extract features of 2D image data using a ResNet network to obtain 2D image features; the second extraction unit is used to extract features of point cloud data using a KPFCNN network to obtain point cloud features; the determination unit is used to perform feature matching on 2D image features and point cloud features using a Transformer network to obtain feature matching results, and determine the three-dimensional coordinates of the target object based on the feature matching results. The device uses a deep learning method to automatically learn effective features in images and point clouds, and refines features through a Transformer network to improve the distinguishability and robustness of features, making them more resistant to the influence of factors such as illumination and occlusion. It solves the problems of low matching accuracy, poor robustness, and inability to effectively solve the scale ambiguity caused by perspective projection in the 2D-3D correspondence visual positioning method in the prior art.

[0136] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-scale visual positioning method, characterized in that: include: Acquire 2D image data and point cloud data within the monitoring range; A ResNet network is used to extract features of the 2D image data to obtain 2D image features; A KPFCNN network is used to extract features of the point cloud data to obtain point cloud features; A Transformer network is used to perform feature matching on the 2D image features and the point cloud features to obtain feature matching results, and the three-dimensional coordinates of the target object are determined according to the feature matching results.

2. The method according to claim 1, characterized in that: The Transformer network is used to perform feature matching on the image features and the point cloud features to obtain feature matching results, including: The image features are enhanced to obtain enhanced image features, and the point cloud features are enhanced to obtain enhanced point cloud features; Determine an anchor feature matrix according to the enhanced image features, wherein the elements in the anchor feature matrix are the enhanced image features; Determine a memory feature matrix according to the enhanced point cloud features, wherein the elements in the memory feature matrix are the enhanced point cloud features; The Transformer network is used to model the association between the elements in the anchor feature matrix and the elements in the memory feature matrix through an attention mechanism to obtain the feature matching result.

3. The method according to claim 2, characterized in that The image feature is enhanced to obtain an enhanced image feature, and the point cloud feature is enhanced to obtain an enhanced point cloud feature, including: Get the first position information enhancement formula in, is the 2D image feature after position information enhancement, is the 2D image feature, is the pixel coordinate matrix of the downsampled image, φ(·) is the Fourier embedding function; Get the second position information enhancement formula in, is the point cloud feature after location information enhancement. is the point cloud feature, is the point corresponding to the coarsest layer in the KPFCNN network; Using the first position information enhancement formula to enhance the image feature, to obtain the enhanced image feature; The point cloud feature is enhanced using the second position information enhancement formula to obtain the enhanced point cloud feature.

4. The method according to claim 2, characterized in that: Using the Transformer network through the attention mechanism, Modeling the association between the elements in the anchor feature matrix and the elements in the memory feature matrix to obtain the feature matching result includes: Obtain a first projection weight, a second projection weight, and a third projection weight, wherein the first projection weight is a projection weight of a query condition, the second projection weight is a projection weight of a feature key, and the third projection weight is a projection weight of a feature value; Projecting the anchor feature matrix and the memory feature matrix using the first projection weight, the second projection weight, and the third projection weight to obtain a correlation between each element in the anchor feature matrix and each element in the memory feature matrix; The correlation is input into a shallow MLP network to further project the attention features to obtain the feature matching result.

5. The method according to claim 4, characterized in that The anchor feature matrix and the memory feature matrix are projected using the first projection weight, the second projection weight, and the third projection weight to obtain the correlation between each element in the anchor feature matrix and each element in the memory feature matrix, including: Get the query condition calculation formula Q=F A W Q 、Characteristic key calculation formula K=F M W K And the eigenvalue calculation formula V = F M W V , where Q is the query condition, K is the feature key, V is the feature value, and F A is the element in the anchor feature matrix, F M is the element in the memory feature matrix, W Q is the first projection weight, W K is the second projection weight, W V is the third projection weight; Get the attention feature formula Among them, Attention(F A , F M ) is F A and F M , Q is the query condition, K is the feature key, V is the feature value, and d is the number of columns of the anchor feature matrix and the memory feature matrix; The correlation between each element in the anchor feature matrix and each element in the memory feature matrix is ​​determined according to the attention feature formula.

6. The method according to claim 4, characterized in that Obtaining a first projection weight, a second projection weight, and a third projection weight, including: Obtaining an initial first weight, an initial second weight, and an initial third weight; Get the total loss formula of the model L all =L coarse +λL fine , where L all is the total loss of the model, L coarse is the first circle loss, λ is the balance factor, L fine is a second circular loss, the total model loss is related to the initial first weight, the initial second weight and the initial third weight, the first circular loss is a circular loss calculated based on the patch feature distance at a coarse level, the coarse level is a level defined by the overlap ratio of the positive and negative samples based on the block, the second circular loss is a circular loss calculated based on the feature distance of pixels and points at a fine level, the fine level is a positive and negative sample based on the spatial distance of pixels and points defined; According to the model total loss formula, the model total loss is determined, and the initial first weight, the initial second weight and the initial third weight are adjusted to minimize the model total loss, and the initial first weight, the initial second weight and the initial third weight corresponding to the minimum model total loss are determined as the first projection weight, the second projection weight and the third projection weight.

7. The method according to claim 6, characterized in that Before obtaining the total loss formula of the model, the method further includes: Get the circular loss formula Among them, d i is the anchor descriptor, L i is d i The corresponding circumferential loss value, one anchor descriptor corresponds to one circumferential loss value, is the Euclidean distance between the i-th sample and the j-th sample, and are the individual weights of positive and negative pairs, and and is the ratio factor between positive sample pairs and negative sample pairs, △ p ,△ n is the expected distance between positive samples and the expected distance between negative samples; The sum of all the circumferential loss values ​​is determined as the circumferential loss.

8. A multi-scale visual positioning device, characterized in that: include: An acquisition unit, used to acquire 2D image data and point cloud data within the monitoring range; A first extraction unit, configured to extract features of the 2D image data using a ResNet network to obtain 2D image features; A second extraction unit is used to extract features of the point cloud data using a KPFCNN network to obtain point cloud features; The determination unit is used to perform feature matching on the 2D image features and the point cloud features using a Transformer network to obtain feature matching results, and determine the three-dimensional coordinates of the target object according to the feature matching results.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the multi-scale visual positioning method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the multi-scale visual positioning method described in any one of claims 1 to 7.

Images