Cross-modal visual positioning method for low-altitude security scene

Abstract

Embodiments of the present application relate to the technical field of visual positioning, and disclose a cross-modal visual positioning method for a low-altitude security scene, which comprises: extracting high-dimensional features of a query image, and performing hash mapping on the high-dimensional features by using an LSH algorithm to obtain hash codes; performing matching in a corresponding feature library according to the hash codes to determine feature images matched with the query image; determining point cloud data corresponding to each feature image in a point cloud data set of a scene to which the query image belongs based on the positions of the feature images; encoding the feature images and the point cloud data to obtain pairs of point cloud feature agents and image feature agents; determining 3D-2D matching relationships between the point cloud data and the feature images according to a cross-modal corresponding relationship learning model and the pairs of point cloud feature agents and image feature agents; and determining a shooting pose corresponding to the query image based on each 3D-2D matching relationship, so as to quickly realize accurate visual positioning in large-scale point cloud.

Description

Technical Field

[0001] This application relates to the field of visual positioning technology, and in particular to a cross-modal visual positioning method for low-altitude security scenarios. Background Technology

[0002] Visual positioning refers to the process of determining the position (usually three-dimensional coordinates) and orientation (usually attitude or angle) of a camera, video camera, or sensor in the real world by analyzing image or video data. Visual positioning technology plays an important role in fields such as autonomous driving, augmented reality, and intelligent navigation. Traditional visual positioning methods mainly rely on feature point matching and geometric models, but under conditions such as complex scenes, changing lighting, and occlusion, traditional visual positioning methods are easily limited and cannot meet the requirements of high-precision positioning.

[0003] In recent years, the rapid development of deep learning technology has brought new opportunities to visual localization. Deep learning-based visual localization methods can utilize deep learning models such as convolutional neural networks to directly learn feature representations from image data, exhibiting better robustness and generalization ability. However, current deep learning-based visual localization methods still suffer from low localization accuracy and insufficient real-time performance in practical applications.

[0004] Currently, visual localization methods in point cloud maps mainly rely on point cloud and image matching, such as visual localization methods based on 2D-3D matching. Some methods rely on overlapping scenes in point clouds and images to design point cloud and image matching methods, such as point cloud and image matching methods based on feature-based dense correspondence frameworks. Other methods utilize image retrieval to complete visual localization in large point clouds, such as motion sensor-guided cross-temporal six-DOF visual localization methods. However, the inventors of this application have found that the above methods all suffer from low visual localization accuracy and slow visual localization speed when processing large dense point cloud data, failing to meet the requirements of real-time visual localization. Summary of the Invention

[0005] The purpose of this application is to provide a cross-modal visual positioning method for low-altitude security scenarios, which can quickly achieve accurate visual positioning in large-scale point clouds and well meet the speed requirements of real-time visual positioning.

[0006] To address the aforementioned technical problems, embodiments of this application provide a cross-modal visual localization method for low-altitude security scenarios, comprising the following steps: inputting the acquired query image into a pre-trained Siamese network to obtain the high-dimensional features of the query image, and using the LSH algorithm (Locality Sensitive Image). Locality-Sensitive Hash (LSH) is used to hash the high-dimensional features of the query image to obtain the hash code of the query image. The hash code of the query image is then matched against a corresponding feature library to determine the target feature image that matches the query image. The feature library belongs to the same scene as the query image, and it stores the hash codes of several feature images. Based on the position coordinates of each target feature image, a pre-built point cloud lookup function is used to determine the point cloud data corresponding to each target feature image in the point cloud dataset corresponding to the scene of the query image. The target feature images and the point cloud data are encoded to obtain paired point cloud feature proxies and image feature proxies. Based on a pre-trained cross-modal correspondence learning model and the paired point cloud feature proxies and image feature proxies, a 3D-2D matching relationship between the point cloud data and the target feature image is determined. Based on each 3D-2D matching relationship, the RANSAC (Random Sample Consensus) algorithm, the EPnP algorithm, and prior knowledge of the gravity direction are used to determine the shooting pose corresponding to the query image.

[0007] The cross-modal visual positioning method for low-altitude security scenarios provided in this application establishes a correspondence between point sets (point clouds) and pixel blocks (images) through cross-modal learning from coarse to fine images to point clouds. It then refines the correspondence at the point and pixel levels to achieve high-precision matching relationship prediction, thereby enabling accurate visual positioning in large-scale point cloud data. The Siamese network, point cloud lookup function, fusion encoder, and cross-modal correspondence learning model used in the visual positioning process employ shared-parameter image feature extractors and point cloud feature extractors. This results in fast training speed, high training quality, and fast inference speed for each model. Simultaneously, the LSH algorithm, based on hash codes and a feature library, is used for matching similar images. The location information of the matched target feature images is easily obtained, allowing for rapid acquisition of the corresponding point cloud data. The entire visual positioning process is short and fast, making it highly suitable for visual positioning of large-scale point cloud data and well-suited for real-time visual positioning applications.

[0008] In some optional embodiments, the pre-trained Siamese network is trained through the following steps: Data augmentation is performed on each image in a preset image dataset to obtain a corresponding augmented dataset; wherein, the images in the image dataset belong to the same scene, and the data augmentation includes at least random translation, rotation, scaling, mirroring, cropping, and color jitter; the augmented images in the augmented dataset are input into an initial Siamese network built based on ResNet50 to obtain the high-dimensional features of each augmented image, and the Euclidean distance between the high-dimensional features of each augmented image is calculated; based on the Euclidean distance between the high-dimensional features of each augmented image, the initial Siamese network is iteratively trained until convergence using a ternary loss function, and the trained initial Siamese network is used as the pre-trained Siamese network; wherein, the ternary loss function is used to ensure that images at the same coordinates have similar features, and images at different coordinates have dissimilar features. The image augmentation process provides more high-quality training samples for the training of the Siamese network. After deployment, the trained Siamese network can quickly and accurately obtain the high-dimensional features of the query image, further improving the accuracy and speed of visual localization.

[0009] In some optional embodiments, the scene to which each image in the image dataset belongs is the same as the query image. The feature library is obtained through the following steps: taking each image in the image dataset as a feature image and establishing a position coordinate mapping for each feature image; inputting each feature image into the Siamese network to obtain the high-dimensional features of each feature image, and using the LSH algorithm to perform hash mapping on the high-dimensional features of each feature image to obtain the hash code of each feature image; and establishing a feature library based on each hash code and the mapping relationship between each hash code and the feature image. For application scenarios, this application establishes a rich feature library in advance, ensuring that the query image can be matched with similar target feature images during hash code matching, thus ensuring that visual localization can be performed smoothly.

[0010] In some optional embodiments, the step of matching the hash code of the query image in the corresponding feature library to determine the target feature image that matches the query image includes: matching the hash code of the query image in the corresponding feature library, calculating the matching similarity between the hash code of the query image and the hash codes in the feature library; and determining all feature images corresponding to hash codes in the feature library whose matching similarity is greater than a first preset threshold as target feature images that match the query image. To increase fault tolerance, this application sets all feature images corresponding to hash codes whose matching similarity is greater than the first preset threshold as target feature images to participate in subsequent registration and visual positioning stages, further improving the accuracy of visual positioning.

[0011] In some optional embodiments, the step of determining the point cloud data corresponding to each target feature image in the point cloud dataset corresponding to the scene to which the query image belongs, based on the position coordinates of each target feature image and using a pre-constructed point cloud lookup function, includes: traversing each target feature image, and in the point cloud dataset corresponding to the scene to which the query image belongs, obtaining the point cloud data within a circular area centered on the position coordinates of the current target feature image and with a preset search radius corresponding to the point cloud lookup function as the radius, as the point cloud data corresponding to the current target feature image; after traversing each target feature image, the point cloud data corresponding to each target feature image is obtained.

[0012] In some optional embodiments, the step of encoding the target feature image and the point cloud data using a fusion encoder to obtain paired point cloud feature proxies and image feature proxies includes: extracting point features from the point cloud data using a PointNet architecture, and performing farthest point sampling on the point cloud data to obtain a preset number of center points and center features corresponding to each center point, where each center point represents the center of a corresponding local region; assigning each point feature to the nearest center point using a point-to-point grouping strategy to determine the local features of the point cloud at each level; and matching the local features of the point cloud at each level with the global features of the target feature image. After feature concatenation, the data is fed into a multilayer perceptron to determine the weights corresponding to the local point cloud features at each level, resulting in weighted point cloud features at each level. A clustering algorithm is then used to extract image features hierarchically, determining the local image features at each level. These local image features are then concatenated with the global point cloud features of the point cloud data and fed into the multilayer perceptron to determine the weights corresponding to the local image features at each level, resulting in weighted image features at each level. Finally, the weighted point cloud features and weighted image features at each level are upsampled to obtain paired point cloud feature proxies and image feature proxies. This method of obtaining paired point cloud feature proxies and image feature proxies allows for the full learning of cross-modal features at different levels, further improving the accuracy of visual localization.

[0013] In some optional embodiments, determining the 3D-2D matching relationship between the point cloud data and the target feature image based on the pre-trained cross-modal correspondence learning model and the paired point cloud feature proxies and image feature proxies includes: inputting the paired point cloud feature proxies and image feature proxies into the cross-modal correspondence learning model; calculating the pairwise distance matrix between the paired point cloud feature proxies and image feature proxies through the coarse matching module of the cross-modal correspondence learning model; obtaining a mask matrix for representing the coarse correspondence based on the pairwise distance matrix; resampling the mask matrix through the coarse-to-fine module of the cross-modal correspondence learning model to obtain point-level features and pixel-level features; and performing fine registration of the point-level features and pixel-level features through the fine matching module of the cross-modal correspondence learning model to obtain the 3D-2D matching relationship between the point cloud data and the target feature image.

[0014] In some optional embodiments, the calculation of the pairwise distance matrix between the pairwise point cloud feature proxies and the image feature proxies is achieved by the following formula:

[0015]

[0016] Among them, f img f represents the point cloud feature proxy. pc The image feature proxy, d p Let D be the preset dimension of the point cloud features, and let D represent the calculated pairwise distance matrix between the paired point cloud feature proxies and the image feature proxies.

[0017] In some optional embodiments, determining the shooting pose corresponding to the query image based on each of the 3D-2D matching relationships, using the RANSAC algorithm and the EPnP algorithm, and prior knowledge of the gravity direction, includes: obtaining the shooting pose corresponding to each of the 3D-2D matching relationships using the RANSAC algorithm and the EPnP algorithm respectively; and determining the shooting pose that best conforms to the prior knowledge of the gravity direction among the shooting poses corresponding to each of the 3D-2D matching relationships as the shooting pose corresponding to the query image.

[0018] In some optional embodiments, after determining the shooting pose corresponding to the query image, the method further includes: if a new query image positioning request is received again from the photographer of the query image, determining whether the time interval between the new query image positioning request and the query image positioning request is less than a second preset threshold, and determining whether the moving speed of the photographer is less than a third preset threshold; if the time interval between the new query image positioning request and the query image positioning request is less than the second preset threshold, and the moving speed of the photographer is less than the third preset threshold, then the shooting pose corresponding to the query image is directly used as the shooting pose corresponding to the new query image; if the time interval between the new query image positioning request and the query image positioning request is not less than the second preset threshold, or the moving speed of the photographer is not less than the third preset threshold, then the new query image is input into the Siamese network to obtain the high-dimensional features of the new query image, and the LSH algorithm is used to perform hash mapping on the high-dimensional features of the new query image to obtain the hash code of the new query image. If the time interval between two location requests initiated by the same shooting party is very short and the shooting party's movement speed is very small, it can be assumed that the shooting party has not moved, and the previous visual positioning result can be directly assigned to the current visual positioning request. Otherwise, a new visual positioning process is performed. This approach satisfies the visual positioning requirements while avoiding the waste of computing resources and saving the cost of visual positioning. Attached Figure Description

[0019] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0020] Figure 1 This is a flowchart of a cross-modal visual positioning method for low-altitude security scenarios, provided in one embodiment of this application.

[0021] Figure 2 This is a flowchart of training a twin network provided in one embodiment of this application;

[0022] Figure 3 This is a flowchart of establishing a feature library provided in one embodiment of this application;

[0023] Figure 4 This is a flowchart provided in one embodiment of the present application, which describes the use of a fusion encoder to encode target feature images and point cloud data respectively to obtain pairs of point cloud feature proxies and image feature proxies;

[0024] Figure 5This is a flowchart provided in one embodiment of the present application, which describes the process of determining the 3D-2D matching relationship between point cloud data and target feature image based on a cross-modal correspondence learning model and paired point cloud feature proxies and image feature proxies.

[0025] Figure 6 This is a structural schematic diagram of a cross-modal visual positioning system for low-altitude security scenarios, provided by another embodiment of this application;

[0026] Figure 7 This is a schematic diagram of a simulation result provided by another embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0029] Currently, visual localization methods in point cloud maps mainly rely on point cloud and image matching. For example, J. Li et al., in their paper "J. Li and G. Hee Lee, 'DeepI2P: Image-to-Point Cloud Registration via DeepClassification,'" in Proc. Conference on Computer Vision and Pattern Recognition, 2021, pp. 15955-15964, proposed a DeepI2P method, a visual localization approach based on 2D-3D matching. This method transforms the registration problem into a classification and inverse camera projection optimization problem, i.e., determining whether the projection of each point in the point cloud is inside or outside the camera truncation, thus improving the accuracy of 2D-3D matching. However, this method fails to find correct matching relationships when processing large, dense point cloud data, and is prone to a large number of incorrect matches at the edges of the truncation. These incorrect matches interfere with RANSAC estimation, thereby affecting the visual localization results.

[0030] Some methods rely on overlapping scenes in point clouds and images to design point cloud and image matching methods. For example, S. Ren et al. published a cross-modal matching algorithm, CorrI2P, in the paper "S. Ren, Y. Zeng, J. Hou, and X. Chen, 'CorrI2P: Deep Image-to-PointCloud Registration via Dense Correspondence,' IEEE Transactions on Circuits and Systems for Video Technology, vol 33, 2022, pp. 1198-1208." This method proposes a point cloud and image matching method based on a feature-based dense correspondence framework. This method converts image and point cloud data into a high-dimensional feature space and inputs the resulting features into a symmetric overlapping region detection module to determine the overlapping regions of the image and point cloud. Finally, it uses the features of the overlapping regions to establish a dense 2D-3D correspondence. However, this method includes multiple modules such as feature embedding, symmetric overlapping region detection, and pose estimation, which leads to high computational complexity, especially when processing large-scale or dense point cloud data. Moreover, if the feature quality is low or there is noise in the image and point cloud, it is likely to lead to a decrease in matching accuracy, especially in complex scenes or low-texture areas, where feature extraction and matching face huge challenges.

[0031] Other methods utilize image retrieval to perform visual localization in large point clouds. For example, S. Yan et al. disclosed SensLoc, a mobile sensor-guided cross-terminal six-DOF visual localization method, in their paper "S. Yan et al., 'Long-term Visual Localization with Mobile Sensors,'" in Proc. Conference on Computer Vision and Pattern Recognition, 2023, pp. 17245-17255. This method utilizes additional sensors on mobile devices to provide initial pose and effective constraints for visual localization, reducing the search space in image matching and final pose estimation. However, in some scenarios, the device may not be able to obtain sufficient sensor data. Sensor data may also be interfered with, affecting the accuracy of localization. Furthermore, the method's versatility and scalability are limited, especially when adapting to various devices, scenarios, and application requirements.

[0032] The methods described above all suffer from low visual positioning accuracy and slow visual positioning speed when processing large-scale dense point cloud data, and cannot meet the requirements of real-time visual positioning.

[0033] For large-scale environments, such as city streets or highways, the amount of point clouds and images can be enormous. Point cloud data is even denser and contains a lot of detailed information. These challenges can lead to a decrease in the accuracy of localization and environmental modeling, and also affect the real-time performance and stability of visual localization systems.

[0034] Because the high-dimensional feature representation methods in 2D images and 3D point clouds are different, and 3D data is denser, while point cloud and image data patterns have visual differences, the similarity between a single point and an image pixel is often fuzzy. It remains difficult to establish the connection between these two feature space descriptors and to accurately obtain the correspondence between pixels and point clouds.

[0035] To address the aforementioned technical problems of low visual positioning accuracy and slow speed, which fail to meet real-time visual positioning requirements, one embodiment of this application proposes a cross-modal visual positioning method for low-altitude security scenarios, applied to a server. The implementation details of the cross-modal visual positioning method for low-altitude security scenarios proposed in this embodiment are described below. These details are provided for ease of understanding and are not essential for implementing this solution.

[0036] The specific process of the cross-modal visual positioning method for low-altitude security scenarios proposed in this embodiment can be described as follows: Figure 1 As shown, it includes:

[0037] Step 101: Input the obtained query image into the pre-trained Siamese network to obtain the high-dimensional features of the query image, and use the LSH algorithm to perform hash mapping on the high-dimensional features of the query image to obtain the hash code of the query image.

[0038] In the specific implementation, the server obtains the location request sent by the photographer in real time. The location request includes a query image x that needs to be visually located. q The server will query image x q Input to the pre-trained Siamese network F twin Twin network F twin Output query image x q The high-dimensional features of the query image x are then processed by the server using the LSH algorithm. q The high-dimensional features are hashed to query the image x. q The high-dimensional features are mapped to a low-dimensional space, and hashed in the low-dimensional space to obtain the query image x. q The hash code.

[0039] In one example, the pre-trained Siamese network F twin It can be controlled by the server through, for example Figure 2 The aforementioned steps are obtained through pre-training, specifically including:

[0040] Step 1011: Perform data augmentation on each image in the preset image dataset to obtain the corresponding augmented dataset, wherein each image in the image dataset belongs to the same scene.

[0041] In a specific implementation, the preset image dataset can be represented as X = {x1, x2, ..., x...} n The image dataset contains n images, all belonging to the same scene. To improve the robustness of image matching and localization, as well as visual localization, in complex and dynamic environments, data augmentation can be performed on the training samples of the Siamese network. This involves randomly translating, rotating, scaling, mirroring, cropping, and color-dithering each image in the image dataset X, repeating this process multiple times to generate diverse image variants, thus obtaining the augmented dataset X corresponding to the image dataset X. ′ ={S1,S2,…,S n}, S i ={x i ,x′ i1 ,x′ i2 ,…,x′ ik Augmented dataset X ′ This provides a more reliable data foundation for image matching and localization.

[0042] Step 1012: Input the augmented images in the augmented dataset into the initial Siamese network built based on ResNet50 to obtain the high-dimensional features of each augmented image, and calculate the Euclidean distance between the high-dimensional features of each augmented image.

[0043] In a specific implementation, the server can obtain data from the augmented dataset X. ′ Randomly select enhanced images As training samples, an initial Siamese network was constructed using ResNet50 as the backbone. The server will S a and S b Input to the initial twin network In the process of obtaining the initial twin network Output S a High-dimensional features and S b The high-dimensional features of S are calculated. a High-dimensional features and S b The Euclidean distance between high-dimensional features.

[0044] Step 1013: Based on the Euclidean distance between the high-dimensional features of each enhanced image, the initial Siamese network is iteratively trained using a ternary loss function until convergence, and the trained initial Siamese network is used as a pre-trained Siamese network.

[0045] In the specific implementation, the server trains the initial twin network. At that time, the ternary loss function was used as the initial Siamese network. The loss function is designed to ensure that images at the same coordinates have similar features, while images at different coordinates have significant feature differences. The server uses a ternary loss function to adjust the initial Siamese network based on the Euclidean distance between the high-dimensional features of each augmented image. Perform iterative training until convergence, and then train the initial Siamese network. As a pre-trained Siamese network F twin This training process enables the Siamese network F... twin It can better distinguish images in different scenarios, improve the accuracy and reliability of image matching, and thus enhance the precision of visual positioning.

[0046] Step 102: Match the hash code of the query image in the corresponding feature library to determine the target feature image that matches the query image. The scene to which the feature library belongs is the same as that of the query image. The feature library stores the hash codes of several feature images.

[0047] In the specific implementation, the server obtains the query image x q After obtaining the hash code, the image x can be queried. q The hash code is matched against the corresponding feature library. The feature library stores the hash codes of several feature images. The scene to which the feature library belongs matches the query image x. q For images belonging to the same context, the server determines the target feature images that match the query image from the corresponding feature library. The set of target feature images can be denoted as X. + .

[0048] In one example, the twin network F twin The scene to which each image in the image dataset X belongs during training corresponds to the query image x. q The scenarios are the same. The server can use methods such as... Figure 3 The steps shown in the diagram for building the feature library specifically include:

[0049] Step 1021: Take each image in the image dataset as a feature image and establish the position coordinate mapping of each feature image.

[0050] In the specific implementation, when building the feature library, the server can use each image in the image dataset X as a feature image and establish a position coordinate mapping for each feature image. The information that needs to be collected during the acquisition of the image dataset X includes the GPS information (position coordinate information) of each frame of the image. The acquired images are named using the position coordinate information, which allows for the rapid establishment of the position coordinate mapping for each feature image.

[0051] Step 1022: Input each feature image into the Siamese network to obtain the high-dimensional features of each feature image, and use the LSH algorithm to perform hash mapping on the high-dimensional features of each feature image to obtain the hash code of each feature image.

[0052] In practical implementation, after the server selects the feature images, it can input each feature image into the Siamese network F. twin The high-dimensional features of each feature image are obtained, and the LSH algorithm is used to hash these features to obtain a hash code for each feature image. The principle of the LSH algorithm is to use multiple random hyperplanes to partition the feature space, mapping each feature vector to a binary hash code. Each hyperplane defines a hash function; when a feature vector lies on one side of the hyperplane, the hash function outputs 1, otherwise it outputs 0. This mapping method ensures that images with similar features are mapped to the same or similar hash codes.

[0053] Step 1023: Establish a feature library based on each hash code and the mapping relationship between each hash code and the feature image.

[0054] In the implementation, after obtaining the hash codes of each feature image, the server can build a feature library based on each hash code and its mapping relationship with the feature images. Essentially, the feature library is a hash code lookup table that records the mapping relationship between hash codes and feature images, as well as the position coordinate mapping relationship of the feature images, to facilitate fast and efficient feature retrieval. For specific application scenarios, the server pre-builds a rich feature library to ensure that the query image matches similar target feature images during hash code matching, guaranteeing successful visual localization.

[0055] In one example, the server queries image x q The hash code is matched against the corresponding feature library to calculate the query image x. q The similarity between the hash code and the hash codes stored in the feature library is used to determine all feature images corresponding to hash codes in the feature library with a similarity greater than a first preset threshold, and these feature images are then identified as matching the query image x. q The target feature image is matched. The first preset threshold can be set by those skilled in the art according to actual needs. To increase fault tolerance, the server uses the feature images corresponding to hash codes with a similarity greater than the first preset threshold as target feature images in subsequent registration and visual localization processes, further improving the accuracy of visual localization.

[0056] In one example, steps 101 to 102 can be referred to as the “image matching coarse localization” stage.

[0057] Step 103: Based on the position coordinates of each target feature image, use a pre-built point cloud lookup function to determine the point cloud data corresponding to each target feature image in the point cloud dataset corresponding to the scene to which the query image belongs.

[0058] In the specific implementation, the server pre-designed a point cloud search function F using the KD-tree algorithm. cut The point cloud lookup function F cut It can provide a small area of ​​point cloud data around a given location coordinate. The server iterates through the feature images of each target and queries the image x. q From the point cloud dataset corresponding to the scene, obtain the point cloud lookup function F with the current target feature image's position coordinates as the center. cut The point cloud data within a circular region with a preset search radius is used as the point cloud data corresponding to the current target feature image. After traversing all target feature images, the server can obtain the point cloud data corresponding to each target feature image. This point cloud lookup function can easily obtain a local point cloud map near a specified location, and the result of the point cloud lookup function also includes information on the number of points within the search area.

[0059] In one example, step 103 could be referred to as the “point cloud retrieval” stage.

[0060] Step 104: Use a fusion encoder to encode the target feature image and point cloud data respectively to obtain pairs of point cloud feature proxies and image feature proxies.

[0061] In practical implementation, the fusion encoder, also known as the fusion encoder that combines multimodal global features and multi-level features, can be denoted as F. fusion For any target feature image x i ∈X + and the corresponding point cloud data p i The server uses the fusion encoder F fusion For the target feature image x respectively i and point cloud data p i Encode the pairs of point cloud feature proxies f. img and image feature proxy f pc Among them, f img =F fusion (x i ), f pc =F fusion (p i ).

[0062] In one example, the server uses a fusion encoder to encode the target feature image and point cloud data separately, obtaining pairs of point cloud feature proxies and image feature proxies, which can be obtained through methods such as... Figure 4The steps shown are implemented as follows:

[0063] Step 1041: Extract point features from point cloud data using the Pointnet architecture, and perform farthest point sampling on the point cloud data to obtain a preset number of center points and the center features corresponding to each center point.

[0064] In the specific implementation, for any target feature image x i ∈X + and the corresponding point cloud data p i Using the Pointnet architecture as the backbone network for point cloud data p i Feature encoding is performed. The server first uses the PointNet architecture to extract point features F from the point cloud data. p Simultaneously, the farthest point sampling is performed on the point cloud to obtain a preset number (m) of center points and the corresponding central features of each center point. The center points are denoted as Q = {q1,q2,…,q}. m The center point represents the center of the corresponding local region, and its central characteristics can be represented as F. Q ,

[0065] Step 1042: Use a point-to-point grouping strategy to assign each point feature to the nearest center point to determine the local features of the point cloud at each level.

[0066] In practice, after the server obtains the point features and the center features, it can use a point-to-point grouping strategy to assign each point feature to the nearest center point, thereby determining the local features of the point cloud at each level.

[0067] Step 1043: After connecting the local features of the point cloud at each level with the global features of the target feature image, feed them into the multilayer perceptron to determine the weights corresponding to the local features of the point cloud at each level, and obtain the weighted point cloud features at each level.

[0068] In the specific implementation, after the server obtains the local features of the point cloud at each level, it can connect the local features of the point cloud at each level with the global features of the target feature image and feed them into the multilayer perceptron. The Softmax operator in the multilayer perceptron is used to learn the corresponding attention weights to determine the weights corresponding to the local features of the point cloud at each level. The local features of the point cloud at each level are multiplied by the corresponding weights to obtain the weighted point cloud features at each level.

[0069] Step 1044: Use a clustering algorithm to extract the target feature image in layers and determine the local features of each layer.

[0070] In the specific implementation, for the target feature image, the server performs positional encoding to add two channels. Then, a simplified clustering algorithm is used to group and extract the depth features of the target feature image in layers, so that the pixels of the target feature image have positional information and can be processed as a series of unordered point sets like point cloud data, thereby determining the local features of the image at each level.

[0071] Step 1045: After connecting the local features of the image at each level with the global features of the point cloud data, the data is fed into the multilayer perceptron to determine the weights corresponding to the local features of the image at each level, thus obtaining the weighted image features at each level.

[0072] In the specific implementation, after the server obtains the local features of the image at each level, it can connect the local features of the image at each level with the global features of the point cloud data and feed them into the multilayer perceptron. The Softmax operator in the multilayer perceptron is used to learn the corresponding attention weights to determine the weights corresponding to the local features of the image at each level. The local features of the image at each level are multiplied by the corresponding weights to obtain the weighted image features at each level.

[0073] Step 1046: Upsample the weighted point cloud features and the weighted image features of each level to obtain paired point cloud feature proxies and image feature proxies.

[0074] In the specific implementation, after the server obtains the weighted point cloud features and weighted image features at each level, it can upsample these features separately. This allows the features to fully learn cross-modal features from different levels during upsampling, resulting in paired point cloud feature proxies and image feature proxies that contain cross-modal multi-level fused features. Using these paired point cloud feature proxies and image feature proxies for subsequent registration and visual localization further improves the accuracy of visual localization.

[0075] Step 105: Based on the pre-trained cross-modal correspondence learning model and paired point cloud feature proxies and image feature proxies, determine the 3D-2D matching relationship between point cloud data and target feature image.

[0076] In its implementation, after obtaining paired point cloud feature proxies and image feature proxies, the server can determine the 3D-2D matching relationship between the point cloud data and the target feature image based on the pre-trained cross-modal correspondence learning model and the paired point cloud feature proxies and image feature proxies. The full name of the "cross-modal correspondence learning model" is "coarse-to-fine sampling cross-modal correspondence learning model," denoted as F. a The server will proxy the paired point cloud features f img and image feature proxy f pcInput into cross-modal correspondence learning model F a In this way, point cloud data p can be obtained. i With target feature image x i 3D-2D matching relationship between them.

[0077] In one example, the server determines the 3D-2D matching relationship between point cloud data and target feature images based on a pre-trained cross-modal correspondence learning model and paired point cloud feature proxies and image feature proxies. This can be achieved through methods such as... Figure 5 The steps shown are implemented as follows:

[0078] Step 1051: Input the paired point cloud feature proxies and image feature proxies into the cross-modal correspondence learning model. Through the coarse matching module of the cross-modal correspondence learning model, calculate the pairwise distance matrix between the paired point cloud feature proxies and image feature proxies, and obtain the mask matrix for representing the coarse correspondence based on the pairwise distance matrix.

[0079] In the specific implementation, after the server obtains paired point cloud feature proxies and image feature proxies, it inputs these pairs of proxies into the cross-modal correspondence learning model. The model's coarse matching module uses the two proxies as input to calculate the pairwise distance matrix between the point cloud and image feature proxies. Based on this distance matrix, a mask matrix representing the coarse correspondence is obtained. The principle is to associate each point set with a pixel patch, projecting 3D points into the corresponding field of view. By calculating the pairwise distance matrix to measure the distance between proxies, the mask matrix is ​​learned.

[0080] In one example, the server calculates the pairwise distance matrix between pairwise point cloud feature proxies and image feature proxies using the following formula:

[0081]

[0082] In the formula, f img f represents point cloud feature proxy pc Represents image feature proxy, d p Let D be the dimension of the preset point cloud features, and let D represent the pairwise distance matrix between the calculated pairwise point cloud feature proxies and image feature proxies.

[0083] Step 1052: The mask matrix is ​​resampled through the coarse-to-fine module of the cross-modal correspondence learning model to obtain point-level features and pixel-level features.

[0084] Step 1053: Through the fine matching module of the cross-modal correspondence learning model, fine registration is performed on point-level features and pixel-level features to obtain the 3D-2D matching relationship between point cloud data and target feature image.

[0085] In its implementation, after the cross-modal correspondence learning model learns the mask matrix through the coarse matching module, it can resample the mask matrix using the coarse-to-fine module to obtain point-level and pixel-level features. Finally, the fine matching module performs fine registration on the point-level and pixel-level features to obtain the 3D-2D matching relationship between the point cloud data and the target feature image, which is the point-to-pixel correspondence.

[0086] Step 106: Based on each 3D-2D matching relationship, the RANSAC algorithm and EPnP algorithm, as well as prior knowledge of the gravity direction, are used to determine the shooting pose corresponding to the query image.

[0087] In the specific implementation, after the server obtains each 3D-2D matching relationship, it can use the RANSAC algorithm and the EPnP algorithm to obtain the shooting pose corresponding to each 3D-2D matching relationship. Then, among the shooting poses corresponding to each 3D-2D matching relationship, the shooting pose that best conforms to the prior knowledge of the gravity direction is determined as the shooting pose corresponding to the query image, thus completing the visual localization.

[0088] In one example, steps 104 to 106 can be referred to as the "image point cloud registration" stage.

[0089] In one example, after completing the visual localization of the query image, if the server receives a new localization request for the query image from the photographer, it can determine whether the time interval between the new localization request and the original query image's localization request is less than a second preset threshold, and also whether the photographer's movement speed is less than a third preset threshold. If the time interval is less than the second preset threshold and the photographer's movement speed is less than the third preset threshold, the server directly uses the shooting pose corresponding to the query image as the shooting pose corresponding to the new query image, completing the visual localization of the new query image. If the time interval is greater than or equal to the second preset threshold, or the photographer's movement speed is greater than or equal to the third preset threshold, the server inputs the new query image into the Siamese network to obtain its high-dimensional features, and uses the LSH algorithm to hash these features to obtain the hash code of the new query image, thus entering a new round of visual localization and achieving visual localization of the new query image.

[0090] Understandably, if the time interval between two location requests initiated by the same shooting party is very short and the shooting party's movement speed is very small, it can be assumed that the shooting party has not moved, and the previous visual positioning result can be directly assigned to the current visual positioning request. Otherwise, a new visual positioning process is performed. This approach satisfies the visual positioning requirements while avoiding the waste of computing resources and saving the cost of visual positioning.

[0091] In this embodiment, cross-modal learning from coarse to fine image to point cloud first establishes the correspondence between point sets (point clouds) and pixel blocks (images), and then refines it at the point level and pixel level to achieve high-precision matching relationship prediction, thereby achieving accurate visual localization in large-scale point cloud data. The Siamese network, point cloud lookup function, fusion encoder, and cross-modal correspondence learning model used in the visual localization process utilize shared parameter image feature extractors and point cloud feature extractors. This results in fast training speed, high training quality, and fast inference speed for each model. Simultaneously, the LSH algorithm is employed, using hash codes and corresponding feature libraries to match similar images. The location information of the matched target feature images is easily obtained, enabling the corresponding point cloud data to be acquired quickly. The entire visual localization process is short and fast, making it highly suitable for visual localization of large-scale point cloud data and well-suited for real-time visual localization applications.

[0092] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0093] Another embodiment of this application proposes a cross-modal visual positioning system for low-altitude security scenarios. The implementation details of the cross-modal visual positioning system for low-altitude security scenarios proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0094] The structural schematic diagram of the cross-modal visual positioning system for low-altitude security scenarios proposed in this embodiment can be seen as follows: Figure 6 As shown, it specifically includes:

[0095] The image matching coarse localization module 201 is used to input the acquired query image into a pre-trained Siamese network to obtain the high-dimensional features of the query image, and use the LSH algorithm to perform hash mapping on the high-dimensional features of the query image to obtain the hash code of the query image. Based on the hash code of the query image, it performs matching in the corresponding feature library to determine the target feature image that matches the query image. The scene to which the feature library belongs is the same as that of the query image, and the feature library stores the hash codes of several feature images.

[0096] The point cloud retrieval module 202 is used to determine the point cloud data corresponding to each target feature image in the point cloud dataset corresponding to the scene to which the query image belongs, based on the position coordinates of each target feature image and using a pre-built point cloud lookup function.

[0097] The image point cloud registration module 203 is used to encode the target feature image and point cloud data respectively using a fusion encoder to obtain pairs of point cloud feature proxies and image feature proxies. Based on the pre-trained cross-modal correspondence learning model and the pairs of point cloud feature proxies and image feature proxies, the 3D-2D matching relationship between the point cloud data and the target feature image is determined. Based on each 3D-2D matching relationship, the RANSAC algorithm and EPnP algorithm, as well as prior knowledge of the gravity direction, are used to determine the shooting pose corresponding to the query image.

[0098] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0099] To verify the superiority of the cross-modal visual positioning method for low-altitude security scenarios proposed in this application, a series of simulation experiments are conducted below for further explanation.

[0100] 1. Simulation conditions

[0101] This experiment was conducted using Python software and the PyTorch deep learning framework on an operating system consisting of an Intel(R) Xeon(R) Silver 4214R CPU, 128GB of memory, two NVIDIA GeForce RTX 3090 GPUs, and Ubuntu 20.04.6. The dataset used in the simulation was a self-collected dataset.

[0102] 2. Simulation Content

[0103] The experiment used a moving vehicle to make continuous visual localization requests. The localization simulation results are as follows: Figure 7 As shown, the center point coordinates represent the vehicle's coordinates in the EGSP:4548 projected coordinate system after subtracting the center point. Figure 7 The dot in the upper right corner represents the vehicle's actual position, the dot on the left represents the vehicle localization result obtained by the cross-modal visual localization method (OURS) for low-altitude security scenarios proposed in this application, and the dot in the lower right corner represents the vehicle localization result using the image retrieval method. Error analysis was performed on the localization results in this experiment, and the results are shown in Table 1, where RTE represents the root mean square error between the predicted position and the actual position, and RRE represents the root mean square error of the relative pose.

[0104] Table 1: Error Analysis of Positioning Results

[0105] Image retrieval methods 3.274952 4.670327 OURS 2.090758 2.681784

[0106] from Figure 7 As can be seen, the results obtained by OURS are basically consistent with the actual driving trajectory of the vehicle, thus proving that OURS can effectively and accurately locate the vehicle in large-scale point clouds. Figure 7 As shown in the lower right corner, OURS calls image matching for coarse localization with a very small number of points, effectively demonstrating the effectiveness of OURS's short-circuit strategy. During the test, the vehicle's localization request was 30 times / second, proving that OURS is suitable for scenarios with real-time localization requirements. Table 1 shows that OURS's localization error is mainly distributed within a 3m range, and the angle error is mainly distributed within a 3° range. Furthermore, the coarse-to-fine sampling strategy effectively reduces the localization error, demonstrating that OURS can perform real-time accurate target localization in large-scale point clouds.

[0107] Another embodiment of this application relates to an electronic device, such as... Figure 8 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions executable by the at least one processor 301, the instructions being executed by the at least one processor 301 to enable the at least one processor 301 to execute the cross-modal visual positioning method for low-altitude security scenarios in the above embodiments.

[0108] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0109] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0110] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0111] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A cross-modal visual positioning method for low-altitude security scenarios, characterized in that, include: The obtained query image is input into a pre-trained Siamese network to obtain the high-dimensional features of the query image, and the LSH algorithm is used to perform hash mapping on the high-dimensional features of the query image to obtain the hash code of the query image. The hash code of the query image is used to match the corresponding feature library to determine the target feature image that matches the query image; wherein the scene to which the feature library belongs is the same as the query image, and the feature library stores the hash codes of several feature images; Based on the position coordinates of each target feature image, a pre-constructed point cloud lookup function is used to determine the point cloud data corresponding to each target feature image in the point cloud dataset corresponding to the scene to which the query image belongs; The target feature image and the point cloud data are encoded using a fusion encoder to obtain pairs of point cloud feature proxies and image feature proxies. Based on the pre-trained cross-modal correspondence learning model and the paired point cloud feature proxies and image feature proxies, the 3D-2D matching relationship between the point cloud data and the target feature image is determined; Based on the 3D-2D matching relationships, the shooting pose corresponding to the query image is determined using the RANSAC algorithm, the EPnP algorithm, and prior knowledge of the gravity direction.

2. The cross-modal visual positioning method for low-altitude security scenarios according to claim 1, characterized in that, The pre-trained Siamese network is obtained through the following steps: Data augmentation is performed on each image in a preset image dataset to obtain a corresponding augmented dataset; wherein, each image in the image dataset belongs to the same scene, and the data augmentation includes at least random translation, rotation, scaling, mirroring, cropping, and color jitter; The enhanced images in the enhanced dataset are input into an initial Siamese network built based on ResNet50 to obtain the high-dimensional features of each enhanced image, and the Euclidean distance between the high-dimensional features of each enhanced image is calculated. Based on the Euclidean distance between the high-dimensional features of each of the enhanced images, the initial Siamese network is iteratively trained until convergence using a ternary loss function, and the trained initial Siamese network is used as a pre-trained Siamese network; wherein, the ternary loss function is used to ensure that images under the same coordinates have similar features, and images under different coordinates have dissimilar features.

3. The cross-modal visual positioning method for low-altitude security scenarios according to claim 2, characterized in that, The scene to which each image in the image dataset belongs is the same as the query image. The feature library is obtained through the following steps: Each image in the image dataset is used as a feature image, and a position coordinate mapping for each feature image is established. Each of the aforementioned feature images is input into the Siamese network to obtain the high-dimensional features of each feature image, and the LSH algorithm is used to perform hash mapping on the high-dimensional features of each feature image to obtain the hash code of each feature image; A feature library is established based on each hash code and the mapping relationship between each hash code and the feature image.

4. The cross-modal visual positioning method for low-altitude security scenarios according to claim 3, characterized in that, The step of matching the hash code of the query image in the corresponding feature library to determine the target feature image that matches the query image includes: The hash code of the query image is matched against the corresponding feature library, and the similarity between the hash code of the query image and the hash code in the feature library is calculated. All feature images in the feature library that correspond to hash codes with a matching similarity greater than a first preset threshold are identified as target feature images that match the query image.

5. The cross-modal visual positioning method for low-altitude security scenarios according to claim 1, characterized in that, The step of determining the point cloud data corresponding to each target feature image in the point cloud dataset corresponding to the scene to which the query image belongs, based on the position coordinates of each target feature image and using a pre-constructed point cloud lookup function, includes: Traverse each target feature image, and in the point cloud dataset corresponding to the scene to which the query image belongs, obtain the point cloud data within a circular area centered on the position coordinates of the current target feature image and with the preset search radius corresponding to the point cloud lookup function as the radius, and use it as the point cloud data corresponding to the current target feature image. After traversing all the target feature images, the point cloud data corresponding to each target feature image is obtained.

6. The cross-modal visual positioning method for low-altitude security scenarios according to claim 1, characterized in that, The step of encoding the target feature image and the point cloud data using a fusion encoder to obtain pairs of point cloud feature proxies and image feature proxies includes: The PointNet architecture is used to extract point features from the point cloud data, and the farthest point sampling is performed on the point cloud data to obtain a preset number of center points and the center features corresponding to each center point. The center point represents the center of the corresponding local area. A point-to-point grouping strategy is adopted to assign each point feature to the nearest center point to determine the local features of the point cloud at each level; After connecting the local features of the point cloud at each level with the global features of the target feature image, the data are fed into a multilayer perceptron to determine the weights corresponding to the local features of the point cloud at each level, thus obtaining the weighted point cloud features at each level. Clustering algorithms are used to extract the target feature image in layers to determine the local features of each layer. After connecting the local features of the image at each level with the global features of the point cloud data, the data is fed into a multilayer perceptron to determine the weights corresponding to the local features of the image at each level, thus obtaining the weighted image features at each level. Upsampling is performed on the weighted point cloud features and the weighted image features of each level to obtain paired point cloud feature proxies and image feature proxies.

7. The cross-modal visual positioning method for low-altitude security scenarios according to claim 1, characterized in that, The step of determining the 3D-2D matching relationship between the point cloud data and the target feature image based on the pre-trained cross-modal correspondence learning model and the paired point cloud feature proxies and image feature proxies includes: The paired point cloud feature proxies and image feature proxies are input into the cross-modal correspondence learning model. The pairwise distance matrix between the paired point cloud feature proxies and image feature proxies is calculated through the coarse matching module of the cross-modal correspondence learning model. Based on the pairwise distance matrix, a mask matrix for representing the coarse correspondence is obtained. The mask matrix is ​​resampled using the coarse-to-fine module of the cross-modal correspondence learning model to obtain point-level features and pixel-level features. The fine matching module of the cross-modal correspondence learning model performs fine registration on the point-level features and pixel-level features to obtain the 3D-2D matching relationship between the point cloud data and the target feature image.

8. The cross-modal visual positioning method for low-altitude security scenarios according to claim 7, characterized in that, The calculation of the pairwise distance matrix between the paired point cloud feature proxies and the image feature proxies is achieved by the following formula: Among them, f img f represents the point cloud feature proxy. pc The image feature proxy, d p Let D be the preset dimension of the point cloud features, and let D represent the calculated pairwise distance matrix between the paired point cloud feature proxies and the image feature proxies.

9. The cross-modal visual positioning method for low-altitude security scenarios according to any one of claims 1 to 8, characterized in that, The process of determining the shooting pose corresponding to the query image based on the 3D-2D matching relationships, using the RANSAC algorithm, the EPnP algorithm, and prior knowledge of the gravity direction, includes: Based on each of the 3D-2D matching relationships, the RANSAC algorithm and the EPnP algorithm are used to obtain the shooting pose corresponding to each of the 3D-2D matching relationships; Among the shooting poses corresponding to each of the 3D-2D matching relationships, the shooting pose that best conforms to the prior knowledge of the direction of gravity is determined as the shooting pose corresponding to the query image.

10. The cross-modal visual positioning method for low-altitude security scenarios according to any one of claims 1 to 8, characterized in that, After determining the shooting pose corresponding to the query image, the method further includes: If a new location request for the query image is received again from the photographer of the query image, it is determined whether the time interval between the new location request and the location request of the query image is less than a second preset threshold, and whether the moving speed of the photographer is less than a third preset threshold. If the time interval between the location request of the new query image and the location request of the query image is less than a second preset threshold, and the moving speed of the photographer is less than a third preset threshold, then the shooting pose corresponding to the query image is directly used as the shooting pose corresponding to the new query image. If the time interval between the location request of the new query image and the location request of the query image is not less than a second preset threshold, or the moving speed of the photographer is not less than a third preset threshold, then the new query image is input into the twin network to obtain the high-dimensional features of the new query image, and the LSH algorithm is used to perform hash mapping on the high-dimensional features of the new query image to obtain the hash code of the new query image.

Images