Unsupervised cross-view image geographic positioning method based on graph structure

Abstract

The invention discloses an unsupervised cross-view-angle image geographic positioning method based on a graph structure, and the method comprises the steps: obtaining a to-be-positioned first view angle image and a reference second view angle image, extracting a to-be-positioned original image feature vector and a reference second view angle image feature vector, and mapping the to-be-positioned original image feature vector and the reference second view angle image feature vector to a same feature space; inputting a to-be-positioned mapping feature vector into the feature map for map search, determining a target path with a starting point as the first mapping feature and an end point as the third mapping feature, and obtaining a third mapping feature vector; and determining a third view angle image containing geographic position information according to the third mapping feature vector, and finally obtaining an image geographic positioning result of the to-be-positioned first view angle image. The method has the advantages that multi-view image fusion is utilized, the labeling cost is reduced through unsupervised learning, the feature relation is mined by means of the graph structure, the geographic positioning accuracy is improved, the method can adapt to various scenes, and the generalization ability is high.

Description

Technical Field

[0001] The present application belongs to the field of computer vision image retrieval, and specifically relates to an unsupervised cross-view image geolocation method based on a graph structure. Background Art

[0002] With the rapid development and widespread application of remote sensing sensors, the number of earth observation images has increased rapidly. Among them, there are many types of ground images and drone images, but most images lack geographic location information. At the same time, remote sensing satellite images usually contain detailed geographic location information, which can provide users with location references for objects. However, due to the large difference in perspective between ground, drone and remote sensing satellite images, the features of the same object under different perspectives are significantly different, which makes it difficult to correlate images from different perspectives. Therefore, exploring a method that can quickly and effectively achieve cross-perspective image geolocation is a problem that needs to be solved urgently in the field of computer vision image retrieval.

[0003] Cross-view geolocation of remote sensing images can be divided into supervised cross-view image positioning and unsupervised cross-view image positioning according to whether true value labels are used. The former is to use the category labels of ground images and satellite images as the basis for supervised training to find the satellite image corresponding to the ground image. The latter is to find the spatial correlation of images from different perspectives without label information, so as to link images from different perspectives.

[0004] The current cross-view remote sensing image localization methods based on deep learning have the following two characteristics: (1) Most cross-image localization methods train models based on the premise that the true value labels of images from different viewpoints are known; (2) In the cross-localization of ground and satellite images, generally only image transformation is performed on the two images for matching. However, in practical applications, there are a large number of unlabeled ground-satellite image pairs; if only labels are relied on for cross-view localization, a lot of human resources will be occupied and the localization efficiency will be low. If ground-satellite image localization is performed only through image transformation, the original spatial features of the image will be distorted and spatial information will be lost during the image transformation process. In this context, it is necessary to study unsupervised ground-remote sensing image cross-view localization methods based on graph structures. Summary of the invention

[0005] The technical purpose of the present application is to provide an unsupervised cross-view image geolocation method based on a graph structure to address the technical problem that the existing cross-view image positioning technology may cause distortion of the original spatial features of the image and loss of spatial information during the image transformation process.

[0006] In order to achieve the above technical objectives, this application adopts the following technical solutions.

[0007] The present application embodiment provides an unsupervised cross-view image geolocation method based on a graph structure, comprising:

[0008] Obtain the first-view image to be positioned, and extract the feature vector of the original image to be positioned;

[0009] Taking the second-view image feature vector extracted from the second-view image as a reference, mapping the second-view image feature vector and the feature vector of the original image to be located to the same feature space;

[0010] Inputting the mapped feature vector to be located after mapping the feature vector of the original image to be located into the feature map obtained by pre-training to perform a graph search of the target path to determine a third mapped feature vector, wherein the starting point of the target path is the first mapped feature and the end point is the third mapped feature;

[0011] Based on the third mapping feature vector, determining a corresponding third perspective image, wherein the third perspective image includes image geographic location information;

[0012] Determine the geographic positioning result of the first-view image to be positioned according to the image geographic location information;

[0013] Among them, the graph nodes of the feature graph include a first mapping feature, a reference mapping feature and a third mapping feature in the same feature space, and the edges of the feature graph include an edge between the first mapping feature and the reference mapping feature, and an edge between the reference mapping feature and the third mapping feature; each of the edges has a corresponding weight; the first mapping feature, the reference mapping feature and the third mapping feature are determined according to the first perspective image, the second perspective image and the third perspective image, respectively.

[0014] In some possible implementations, the first mapping feature, the reference mapping feature, and the third mapping feature are determined as follows:

[0015] Performing feature extraction on the first perspective image, the second perspective image, and the third perspective image respectively to obtain a first image feature vector, a second image feature vector, and a third image feature vector;

[0016] Determine a reference image feature vector according to the second image feature vector;

[0017] The first image feature vector, the reference image feature vector and the third image feature vector are mapped to the same feature space based on the reference image feature vector, so as to obtain the first mapping feature vector, the reference mapping feature vector and the third mapping feature vector.

[0018] In some possible implementations, the reference image feature vector is the second image feature vector, or the reference image feature vector is each cluster center feature vector obtained by clustering the second image feature vector.

[0019] In some possible implementations, the reference image feature vector is a feature vector of each cluster center obtained by clustering the second image feature vector; each of the reference image features in the reference image feature vector carries a pseudo label of the corresponding cluster;

[0020] Each first mapping feature in the first mapping feature vector is assigned a pseudo label of the corresponding first mapping reference feature vector; the first mapping reference feature vector is determined by: for each first mapping feature, the feature similarity between each reference mapping feature in the reference mapping feature vector and the first mapping feature is arranged in descending order, and a predetermined number of reference mapping features are selected to form the first mapping reference feature vector;

[0021] Each third mapping feature in the third mapping feature vector is assigned a pseudo label of the corresponding unmapped reference feature vector; the unmapped reference feature vector is determined by: for each of the third mapping features, the feature similarities between each reference mapping feature in the reference mapping feature vector and the third mapping feature are arranged in descending order, and the reference mapping features of the set number mentioned above are selected to form the unmapped reference feature vector;

[0022] Inputting the mapped feature vector to be located after mapping the feature vector of the original image to be located into the pre-trained feature map to perform a graph search for the target path to obtain a third mapped feature vector, including:

[0023] A graph search is performed with the goal of minimizing the weighted sum of the edges in the target path and ensuring that the label similarity between the pseudo labels of the nodes at both ends of the target path meets preset requirements, thereby determining the third mapping feature vector.

[0024] In some possible implementations, the method for determining the first mapping reference feature vector specifically includes:

[0025] For each of the first mapping features, determine the vector dot product of the first mapping feature and each of the reference mapping features in the reference mapping feature vector in turn to obtain the similarities of each first feature; arrange the similarities of each of the first features in descending order, and select a predetermined number of reference mapping features to form the first mapping reference feature vector;

[0026] Or, for each of the first mapping features, determine the vector dot product of each of the reference mapping features and the first mapping feature in turn to obtain the similarities of each second feature; arrange the similarities of each of the second features in descending order, and select a predetermined number of reference mapping features to form the first mapping reference feature vector;

[0027] Or, for each of the first mapping features, determine the similarities of each of the first features and each of the second features respectively, and arrange them in descending order; compare the similarities of the first features and the second features in turn, and select the reference mapping feature with the largest value between the two, until the set number of reference mapping features mentioned above are selected to form the first mapping reference feature vector.

[0028] In some possible implementations, the label similarity between the pseudo labels of the nodes at both ends of the target path meets the preset requirements, including:

[0029] Performing a vector dot product operation on the pseudo label of the first mapping feature vector and the pseudo label of the third mapping feature vector to obtain the label similarity;

[0030] When the tag similarity is greater than or equal to the threshold, the preset requirement is met.

[0031] In some possible implementations, the method further comprises:

[0032] For each of the first mapping features, calculating the feature similarity between the first mapping feature and all reference mapping features in the reference mapping vector, and using the feature similarity value as the corresponding association probability;

[0033] For a single first mapping feature, the corresponding reference mapping features are arranged in descending order according to the feature similarity values, a number of reference mapping features with the highest feature similarity to the first mapping feature are selected as positive samples, and negative samples are selected according to the remaining reference mapping features;

[0034] The selected positive samples and negative samples are substituted into a pre-selected contrast loss function, and the contrast loss function is used to calculate the loss value based on the associated probability. The parameters are adjusted by the loss value, so as to reduce the similar image feature distance between the first mapping feature and the reference mapping feature and between the third mapping feature and the reference mapping feature, and increase the dissimilar image feature distance.

[0035] In some possible implementations, negative samples are selected based on the remaining reference mapping features, including:

[0036] All other reference mapping features are used as negative samples, or the negative samples are determined from all other reference mapping features using a difficult sample selection method.

[0037] In some possible implementations, the method further includes: taking the reference mapping feature that ranks first in feature similarity with the first mapping feature as a positive sample, and taking all other reference mapping features as negative samples.

[0038] In some possible implementations, the method further comprises:

[0039] Collecting a first-perspective image, a second-perspective image, and a third-perspective image, all of which take an object located in a target area as a photographing object, to form a multi-perspective image data set, wherein the third-perspective image includes geographic location information of the target area;

[0040] Dividing the multi-view image dataset into a training set and a test set;

[0041] Construct the initial graph structure;

[0042] According to a preset goal, the initial graph structure is trained using the training set, and an optimization algorithm is selected to optimize the initial graph structure until a preset training stop condition is reached;

[0043] The trained initial graph structure is tested using a test set to determine the feature graph.

[0044] Compared with the prior art, the unsupervised cross-view image geolocation method based on graph structure provided in the embodiment of the present application has the following beneficial technical effects: many traditional image geolocation methods often require a large amount of image data with accurate geographic annotation information as supervision information to train the model, while the image geolocation method provided in the present application is unsupervised and does not require tedious geo-annotation of the image in advance. By acquiring images from different perspectives, extracting image features and mapping them to the same feature space, and then operating based on the structure of the feature graph, the hidden associations between images from different perspectives can be displayed in the form of a graph. By using the weight settings of the nodes (first mapping features, reference mapping features, third mapping features) and edges in the feature graph, when performing graph search to determine the target path, the appropriate mapping relationship can be accurately found based on the closeness of the association between the features, and the first-perspective image to be located can be gradually associated with the third-perspective image containing geographic location information, thereby improving the accuracy of the final geolocation result and reducing the positioning error caused by perspective differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are for explanation purposes only and are not intended to limit the scope of the present application in any way. In addition, the shapes and proportional dimensions of the components in the drawings are only for illustration purposes and are used to help understand the present application. They do not specifically limit the shapes and proportional dimensions of the components of the present application. Under the guidance of the present application, those skilled in the art can select various possible shapes and proportional dimensions to implement the present application according to specific circumstances. In the drawings:

[0046] Figure 1 A schematic diagram of the flow of an unsupervised cross-view image geolocation method based on a graph structure provided in an embodiment of the present application;

[0047] Figure 2 This is a schematic diagram of the process of three-view image comparison learning in the embodiment of the present application;

[0048] Figure 3 Schematic diagram of the principle of the unsupervised cross-view image geolocation process in the embodiment. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0050] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features.

[0051] When ground-to-satellite image positioning is traditionally performed only through image transformation, due to the limitations of the image transformation operation itself (such as some geometric transformations, projection transformations, etc.), the original spatial characteristics of the image will be distorted. Spatial features such as the original object shape and relative position relationship will change, and some spatial information will be lost, which will affect the accuracy of subsequent positioning.

[0052] To solve the above technical problems, the present application provides an unsupervised cross-view image geolocation method based on a graph structure, including:

[0053] Step S1, obtaining a first-view image to be positioned;

[0054] Step S2, taking the second-view image feature vector extracted from the second-view image as a reference, mapping the second-view image feature vector and the feature vector of the original image to be located to the same feature space;

[0055] Step S3, inputting the mapped feature vector to be located after mapping the feature vector of the original image to be located into the feature map obtained by pre-training to perform a graph search of the target path to determine the third mapped feature vector, wherein the starting point of the target path is the first mapped feature and the end point is the third mapped feature;

[0056] Step S4: based on the third mapping feature vector, determine the corresponding third-view image, wherein the third-view image includes image geographic location information; and determine the geographic location result of the first-view image to be located according to the image geographic location information.

[0057] In the embodiment, the first perspective image, the second perspective image, and the third perspective image may all be photographed with objects within a set area, and the objects may be the same object or different objects, and the objects may be terrain features, buildings, or street scenes, etc. For example, the first perspective image is a ground image of block A, and the second perspective image and the third perspective image are drone images and satellite images within a set range (satisfying the error requirement) near block A, respectively.

[0058] In an embodiment, the graph nodes of the feature graph include a first mapping feature, a reference mapping feature, and a third mapping feature in the same feature space, and the edges of the feature graph include an edge between the first mapping feature and the reference mapping feature, and an edge between the reference mapping feature and the third mapping feature; each edge has a corresponding weight; the first mapping feature, the reference mapping feature, and the third mapping feature are determined according to the first perspective image, the second perspective image, and the third perspective image, respectively.

[0059] As an example, the first mapping feature, the reference mapping feature and the third mapping feature are determined as follows: feature extraction is performed on the first perspective image, the second perspective image and the third perspective image, respectively, to obtain a first image feature vector, a second image feature and a third image feature vector; the reference image feature vector is determined based on the second image feature vector; the first image feature vector, the reference image feature vector and the third image feature vector are mapped to the same feature space based on the reference image feature vector, so as to obtain a first mapping feature vector, a reference mapping feature vector and a third mapping feature vector.

[0060] The graph search in step S2 can determine the path with the smallest edge weight as the target path.

[0061] In some embodiments, the weight of each edge in the feature graph is determined according to the similarity between the graph nodes at both ends of the edge. As an example, the Euclidean distance, Manhattan distance or Pearson correlation coefficient of the mapping features of the graph nodes at both ends of the edge can be calculated as the similarity between the graph nodes at both ends of the edge.

[0062] For example, for two image feature vectors (assuming they are vector A and vector B with the same dimension), the Euclidean distance calculation formula is: , where n is the dimension of the vector, a i and b i are the i-th elements of vectors A and B respectively. The smaller the distance, the more similar the features of the two images are. Usually a distance threshold is set to determine whether they are similar.

[0063] In some embodiments, the Dijkstra algorithm may be used to perform a graph search on a weighted adjacency matrix to find a path with the maximum similarity, thereby achieving matching between cross-view images. The Dijkstra algorithm is a prior art and will not be described in detail in this application.

[0064] In a specific implementation, the reference image feature vector is the second image feature vector, or the reference image feature vector is the feature vector of each cluster center obtained by clustering the second image feature vector. The data volume of the second-view image feature vector is often large and there may be certain redundant information and noise interference. By extracting the cluster center feature as the reference image feature by clustering, many similar or highly correlated features can be clustered into one category, and the cluster center is used to represent this category of features, which greatly simplifies the data representation, reduces the amount of data to be processed, and weakens the influence of abnormal features caused by noise to a certain extent, which is conducive to improving the efficiency and accuracy of the geolocation process.

[0065] In some embodiments, the reference image feature vector is the feature vectors of each cluster center obtained by clustering the second image feature vector; each reference image feature carries a pseudo-label of the corresponding cluster; after mapping the first image feature vector, the reference image feature vector and the third image feature to the same feature space, the pseudo-label of each reference mapping feature in the obtained reference mapping feature vector remains unchanged.

[0066] Each first mapping feature in the first mapping feature vector is assigned a pseudo label of the corresponding first mapping reference feature vector; the first mapping reference feature vector is determined as follows: for each first mapping feature, the feature similarities between each reference mapping feature in the reference mapping feature vector and the first mapping feature are arranged in descending order, and a predetermined number of reference mapping features are selected to form the first mapping reference feature vector.

[0067] Each third mapping feature in the third mapping feature vector is assigned a pseudo label corresponding to the last mapping reference feature vector; the last mapping reference feature vector is determined as follows: for each third mapping feature, the feature similarities between each reference mapping feature and the third mapping feature in the reference mapping feature vector are arranged in descending order, and a set number of reference mapping features are selected to form the last mapping reference feature vector.

[0068] In some embodiments, the mapped feature vector to be located after mapping the feature vector of the original image to be located is input into the pre-trained feature map to perform a graph search of the target path to determine the third mapped feature vector, including: performing a graph search with the goal of minimizing the weighted sum of the edges in the target path and the label similarity between the pseudo labels of the nodes at both ends of the target path meeting the preset requirements, thereby determining the third mapped feature vector. In some embodiments, not only the weighted sum of the edges of the path is minimized, but also the nodes at both ends of the path must have sufficient similarity in a certain feature classification or attribute.

[0069] As an example, suppose a multi-label vector of the first image map feature vector representing the starting point of the target path, The multi-label vector of the third image mapping feature vector representing the target path endpoint is used to perform a vector dot product operation on the two as the label similarity Y of the two, such as: ,if , it means that the first mapping feature vector and the third mapping feature vector contain at least one same category, and the two must be similar. P is the threshold value, which can be set to 1.

[0070] The target path determined by graph search under dual condition constraints can more accurately associate the features corresponding to images from different perspectives according to reasonable logic, making the entire process from the image to be located to the final determination of geographic location more accurate and reliable, and improving the accuracy of the final geographic location result.

[0071] In some embodiments, the weight of an edge may also be the label similarity between pseudo labels of nodes at both ends of the edge. The label similarity may be determined using methods such as Euclidean distance, Manhattan distance, or Pearson correlation coefficient.

[0072] In some embodiments, Figure 2 As shown, the unsupervised cross-view image geolocation method based on graph structure also includes: using a contrast learning method to reduce the distance between similar image features between the first mapping feature and the reference mapping feature and between the third mapping feature and the reference mapping feature, and increase the distance between dissimilar image features. Taking the first mapping feature as an example, it specifically includes:

[0073] For each first mapping feature, calculate the feature similarity between it and all reference mapping features in the reference mapping vector, and use the feature similarity value as the corresponding association probability;

[0074] For a single first mapping feature, the corresponding reference mapping features are arranged in descending order according to the feature similarity values, and a number of reference mapping features (the specific number can be determined according to a preset selection number or other criteria, such as selecting the first one) with the feature similarity to the first mapping feature are selected as positive samples, and negative samples are selected according to the remaining reference mapping features;

[0075] The selected positive samples and negative samples are substituted into the pre-selected contrast loss function, and the loss value is calculated based on the association probability using the contrast loss function. The parameters are adjusted by the loss value, so that the distance between similar image features between the first mapping feature and the reference mapping feature is reduced, and the distance between dissimilar image features is increased.

[0076] The method of reducing the similar image feature distance between the third image feature and the reference image feature and increasing the dissimilar image feature distance is the same as above and will not be repeated herein.

[0077] In some embodiments, selecting negative samples according to the remaining reference mapping features includes: using all the remaining reference mapping features as negative samples, or determining negative samples from all the remaining reference mapping features using a difficult sample selection method.

[0078] In a specific implementation, step S3 may pre-establish a feature index library, establish an index relationship between the third mapping feature extracted from the third-view image and the corresponding third-view image, and construct a feature index library. The third mapping feature determined by the graph search is used as a query condition, and a matching search is performed in the constructed feature index library. A nearest neighbor search algorithm (such as a K-nearest neighbor algorithm, etc.) may be used to calculate the similarity between the third mapping feature and each third mapping feature stored in the index library (such as using a similarity measurement method such as Euclidean distance, cosine similarity, etc. to determine the similarity), find the most similar third mapping feature stored in the index library, and then determine the corresponding third-view image according to the index relationship.

[0079] In a specific embodiment, the unsupervised cross-view image geolocation method based on a graph structure further includes first constructing an initial graph structure, and training the graph structure to obtain a feature graph. As an example, Figure 3 As shown, the training process includes:

[0080] Step S101: collect a first-view image, a second-view image, and a third-view image to construct a multi-angle image library, wherein the third-view image includes geographic location information of a target area.

[0081] As an example, the first perspective image is a near-earth observation image, such as a ground perspective image; the second perspective image is an aerial observation image, such as a drone perspective image; the third perspective is a high-altitude observation image, such as a satellite perspective image, and the satellite image includes the image's geographic location information. Images from different height perspectives have completely different imaging characteristics and features, providing more abundant and diverse feature data for the unsupervised learning process, so that based on these rich features, the intrinsic connection between images can be better mined, the mapping relationship between different perspective features in the same feature space and the association logic in the graph structure can be learned, so that unsupervised learning can play a more full role, while reducing the annotation cost, it can still accurately achieve geographic positioning.

[0082] The drone-view images in the test set can be divided into a drone-view image set at a first altitude and a drone-view image set at a second altitude according to altitude.

[0083] In the embodiment, the Unversity-1652 dataset can be used. The data of the dataset is collected from three different platforms, namely, a simulated drone platform, satellite images, and street view images. The Unversity-1652 dataset is divided into a training set , test set Two sub-image libraries, one of which is the training set Network training and test sets for graph structures Used for cross-view positioning result evaluation.

[0084] At the initial stage of training, equal amounts of images from three perspectives are randomly selected.

[0085] Step 102: extract image features from the first perspective image, the second perspective image, and the third perspective image, respectively, to obtain corresponding first perspective image feature vectors, second perspective image feature vectors, and third perspective image feature vectors, respectively.

[0086] Due to the different viewing angles, the image features of the three viewing angles are quite different. In a specific embodiment, a three-view convolutional network (three parallel branch convolutional networks) can be constructed to simultaneously process images from three viewing angles.

[0087] In some embodiments, for drone-view images and satellite-view images, since the two images are highly similar and the shooting angles are not much different, when designing a three-view convolutional network for processing, the branch convolutional networks for the two images can share weights.

[0088] As an example, the ResNet residual network can be used for image feature extraction, but the feature extraction method designed in this application is not limited to the ResNet residual network, and other networks can also be used in specific implementations.

[0089] Step 103: cluster the second image feature vector to obtain a cluster center feature vector as the reference image feature vector, perform K-means clustering using all the second view image features in the data set (other clustering methods may also be used to achieve clustering), and finally obtain a cluster center feature vector The unsupervised clustering method based on the features of drone view images is a prior art and will not be described in detail in this application.

[0090] Step 104: Using the reference image feature vector (i.e., the cluster center feature vector obtained in step 104) ) is used as a reference to map the first image feature vector, the reference image feature vector and the third image feature vector to the same feature space to obtain a first mapping feature vector, a reference mapping feature vector and a third mapping feature vector.

[0091] As an example, the specific process of feature mapping includes: assuming represents the first-view image feature vector, represents the third-view image feature vector, Represents the feature vector of the cluster center of the second-view image, that is, the feature vector of the reference image, where , Represents a feature vector of a certain perspective, d0 represents the dimension of the image feature, and d is the feature dimension after mapping. Here, a linear matrix of size d0×d is used, and the parameter is θ (which can be optimized through comparative learning later). The image feature vectors of different perspectives are input into the adapter Θ, and the adapter Θ will process the input feature vector according to its own internal parameters and operation rules, and map it to the same feature space:

[0092] (1)

[0093] Step 105: Based on the mapping feature vectors of different perspectives of the same feature space obtained in step 104, a pseudo label corresponding to the first mapping reference feature vector is assigned to each first mapping feature; the first mapping reference feature vector is determined by arranging the feature similarities between each reference mapping feature in the reference mapping feature vector and the first mapping feature in descending order for each first mapping feature, and selecting a predetermined number of reference mapping features to form the first mapping reference feature vector;

[0094] Each third mapping feature is assigned a pseudo label corresponding to the last mapping reference feature; the last mapping reference feature is determined as follows: for each third mapping feature, the feature similarities between each reference mapping feature and the third mapping feature in the reference mapping feature vector are arranged in descending order, and a set number of reference mapping features are selected to form the last mapping reference feature vector.

[0095] Taking the first mapping feature vector of the first perspective (ground perspective) and the reference mapping feature vector as an example, firstly, for a first mapping feature vector and a reference mapping feature vector of the second perspective (cluster center feature of the drone perspective image) , perform vector dot product operation on the two, including determining the vector dot product of the reference image feature and the first image feature, and obtaining the first feature similarity (As shown in Formula 2); Determine the vector dot product of the first image feature and the reference image feature to obtain the second feature similarity (As shown in Formula 3):

[0096] (2)

[0097] (3)

[0098] Then, assuming that the image mapping features of the two perspectives satisfy the same pseudo-label distribution, the pseudo-labels of the drone perspective images are obtained by unsupervised clustering.

[0099] The method for determining the first mapping reference feature vector specifically includes:

[0100] For each first mapping feature, determine the vector dot product of the first mapping feature and each reference mapping feature in the reference mapping feature vector in turn to obtain the similarity of each first feature; arrange the similarities of each first feature in descending order, and select a predetermined number of reference mapping features to form a first mapping reference feature vector;

[0101] Or, for each first mapping feature, determine the vector dot product of each reference mapping feature and the first mapping feature in turn to obtain the similarity of each second feature; arrange the similarities of each second feature in descending order, and select a predetermined number of reference mapping features to form a first mapping reference feature vector;

[0102] Or, for each first mapping feature, determine the similarity of each first feature and the similarity of each second feature respectively, and arrange them in descending order; compare the similarity of the first feature and the similarity of the second feature in turn, and select the reference mapping feature with the largest value between the two, until the reference mapping features of the preset number are selected to form the first mapping reference feature vector

[0103] Assume that the first mapping feature pseudo-label composition of the ground image is In one embodiment, Optional or Arrange the pseudo labels corresponding to the previously set number (n in total) of drone cluster center features in descending order; or compare or , select the largest drone cluster center feature among the two, until the pseudo labels corresponding to the previously set number (n in total) of drone cluster center features are selected. Pseudo labels can be used for label similarity comparison when performing graph search to determine the target path.

[0104] In some embodiments, the method further includes step 106, using a comparative learning method to bring the image features of the same location closer together and push the image features of different locations farther apart in the first mapping features of the first perspective and the reference mapping features of the second perspective. Figure 2 shown.

[0105] For each first mapping feature, calculate the feature similarity between it and all reference mapping features, and use the feature similarity value as the corresponding association probability (that is, the probability that the ground image mapping feature belongs to a cluster center feature in the drone view image clustering features); for a single first mapping feature, arrange the corresponding reference mapping features in descending order according to the feature similarity value, select a number of reference mapping features with the highest similarity to the first mapping feature (such as the first similarity) as positive samples, and select negative samples based on the remaining reference mapping features (such as all the remaining reference mapping features as negative samples, or use the difficult learning method to select the drone cluster center features corresponding to the first K-1 elements of the pseudo-label as difficult negative samples); substitute the selected positive and negative samples into the pre-selected contrast loss function, use the contrast loss function to calculate the loss value based on the association probability, and adjust the parameters through the loss value, so as to reduce the distance between the first mapping feature and the reference mapping feature in similar image features and increase the distance between dissimilar image features.

[0106] By using the contrast learning method, the method of shortening the distance between the image features of the same location and the image features of different locations in the third mapping features of the third perspective and the reference mapping features of the second perspective is the same as above and will not be repeated. By shortening the distance between the features corresponding to the images of the same location from different perspectives (the first perspective and the second perspective, the third perspective and the second perspective) is able to further strengthen the intrinsic connection between these features from different perspectives but representing the same actual location.

[0107] For the contrastive learning method, the loss function is defined as follows based on the InfoNCE loss function:

[0108] (4)

[0109] (5)

[0110] (6)

[0111] in, and They represent the asymmetric difficult sample contrast learning loss function from the first-view image (ground image) mapping feature vector / third-view image (satellite image) mapping feature vector to the drone cluster center feature vector (reference mapping feature vector). Ω represents the number of first-view images (ground images) / third-view images (satellite images), and τ is a temperature parameter that controls the smoothness of the similarity distribution. and They represent the i-th first mapping feature and the j-th third mapping feature respectively. and Respectively represent the drone cluster center features closest to the i-th first mapping feature and the j-th third mapping feature as positive samples. N and M represent the K drone cluster center negative sample sets that are most similar to the i-th first mapping feature or the j-th third mapping feature.

[0112] Step 107: construct an initial graph structure G with the final first mapping feature, reference mapping feature (UAV view image cluster center feature) and third mapping feature obtained in step 106. The graph G nodes include three types: ground image mapping feature (first mapping feature) node G i , UAV image mapping feature (reference mapping feature) node D j and satellite image mapping feature (third mapping feature) node S k Each node is associated with a feature vector obtained from the feature extraction network. i The eigenvector of is f(G i ), UAV image mapping feature node D j The eigenvector of is f(D j ), satellite image mapping feature node S k The eigenvector of is f(S k ).

[0113] Then, the similarities between the ground image mapping feature nodes and the UAV image mapping feature nodes, as well as the satellite image mapping feature nodes and the UAV image mapping feature nodes are calculated as the edge weights.

[0114] Assume that the ground image maps feature nodes G i and drone image mapping feature node D j The edge weight between , the edge weight between the satellite image mapping feature node and the drone image mapping feature node is . Both are calculated as follows:

[0115] (7)

[0116] (8)

[0117] If two nodes are adjacent to each other, then the edge exists, otherwise, the edge does not exist. is the weighted adjacency matrix, calculated as follows:

[0118] (9)

[0119] According to the preset goal, the initial graph structure is trained using the training set, and an optimization algorithm is selected to optimize the initial graph structure until the preset training stop condition is reached; the trained initial graph structure is tested using the test set to determine the feature graph.

[0120] The method provided in this application uses multi-view images, which can integrate information of different scales, achieve a balance between details and the overall situation, and improve positioning accuracy. Based on unsupervised learning, it reduces the burden of data annotation and enhances generalization ability. With the help of graph structure and its edge weights, node pseudo-labels, etc., it deeply explores feature associations, ensures reliable positioning, and is suitable for diverse geographical scenarios.

[0121] The above is a detailed introduction to the graph-based unsupervised cross-view image geolocation method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the concept of the present application and should not be construed as limiting the scope of protection of the present application.

Claims

1. An unsupervised cross-view image geolocation method based on graph structure, characterized in that: include: Obtain the first-view image to be positioned, and extract the feature vector of the original image to be positioned; Taking the second-view image feature vector extracted from the second-view image as a reference, mapping the second-view image feature vector and the feature vector of the original image to be located to the same feature space; Inputting the mapped feature vector to be located after mapping the feature vector of the original image to be located into the feature map obtained by pre-training to perform a graph search of the target path to determine a third mapped feature vector, wherein the starting point of the target path is the first mapped feature and the end point is the third mapped feature; Based on the third mapping feature vector, determining a corresponding third perspective image, wherein the third perspective image includes image geographic location information; Determine the geographic positioning result of the first-view image to be positioned according to the image geographic location information; The graph nodes of the feature graph include a first mapping feature, a reference mapping feature, and a third mapping feature in the same feature space, and the edges of the feature graph include an edge between the first mapping feature and the reference mapping feature, and an edge between the reference mapping feature and the third mapping feature; Each of the edges has a corresponding weight; the first mapping feature, the reference mapping feature and the third mapping feature are determined according to the first perspective image, the second perspective image and the third perspective image respectively.

2. The unsupervised cross-view image geolocation method based on graph structure according to claim 1, characterized in that: The first mapping feature, the reference mapping feature and the third mapping feature are determined as follows: Performing feature extraction on the first perspective image, the second perspective image, and the third perspective image respectively to obtain a first image feature vector, a second image feature vector, and a third image feature vector; Determine a reference image feature vector according to the second image feature vector; The first image feature vector, the reference image feature vector and the third image feature vector are mapped to the same feature space based on the reference image feature vector, so as to obtain the first mapping feature vector, the reference mapping feature vector and the third mapping feature vector.

3. The unsupervised cross-view image geolocation method based on graph structure according to claim 2, characterized in that: The reference image feature vector is the second image feature vector, or the reference image feature vector is each cluster center feature vector obtained by clustering the second image feature vector.

4. The unsupervised cross-view image geolocation method based on graph structure according to claim 2, characterized in that: The reference image feature vector is a feature vector of each cluster center obtained by clustering the second image feature vector; each reference image feature in the reference image feature vector carries a pseudo label of the corresponding cluster; Each first mapping feature in the first mapping feature vector is assigned a pseudo label of the corresponding first mapping reference feature vector; the first mapping reference feature vector is determined by: for each first mapping feature, the feature similarity between each reference mapping feature in the reference mapping feature vector and the first mapping feature is arranged in descending order, and a predetermined number of reference mapping features are selected to form the first mapping reference feature vector; Each third mapping feature in the third mapping feature vector is assigned a pseudo label of the corresponding unmapped reference feature vector; the unmapped reference feature vector is determined by: for each of the third mapping features, the feature similarities between each reference mapping feature in the reference mapping feature vector and the third mapping feature are arranged in descending order, and the reference mapping features of the set number mentioned above are selected to form the unmapped reference feature vector; Inputting the mapped feature vector to be located after mapping the feature vector of the original image to be located into the pre-trained feature map to perform a graph search for the target path to obtain a third mapped feature vector, including: A graph search is performed with the goal of minimizing the weighted sum of the edges in the target path and ensuring that the label similarity between the pseudo labels of the nodes at both ends of the target path meets preset requirements, thereby determining the third mapping feature vector.

5. The unsupervised cross-view image geolocation method based on graph structure according to claim 4, characterized in that: The method for determining the first mapping reference feature vector specifically includes: For each of the first mapping features, determine the vector dot product of the first mapping feature and each of the reference mapping features in the reference mapping feature vector in turn to obtain the similarities of each first feature; arrange the similarities of each of the first features in descending order, and select a predetermined number of reference mapping features to form the first mapping reference feature vector; Or, for each of the first mapping features, determine the vector dot product of each of the reference mapping features and the first mapping feature in turn to obtain the similarities of each second feature; arrange the similarities of each of the second features in descending order, and select a predetermined number of reference mapping features to form the first mapping reference feature vector; Or, for each of the first mapping features, determine the similarities of each of the first features and each of the second features respectively, and arrange them in descending order; compare the similarities of the first features and the second features in turn, and select the reference mapping feature with the largest value between the two, until the set number of reference mapping features mentioned above are selected to form the first mapping reference feature vector.

6. The unsupervised cross-view image geolocation method based on graph structure according to claim 4, characterized in that: The label similarity between the pseudo labels of the nodes at both ends of the target path meets the preset requirements, including: Performing a vector dot product operation on the pseudo label of the first mapping feature and the pseudo label of the third mapping feature to obtain the label similarity; When the tag similarity is greater than or equal to the threshold, the preset requirement is met.

7. The unsupervised cross-view image geolocation method based on graph structure according to claim 1, characterized in that: The method further comprises: For each of the first mapping features, calculating the feature similarity between the first mapping feature and all reference mapping features in the reference mapping vector, and using the feature similarity value as the corresponding association probability; For a single first mapping feature, the corresponding reference mapping features are arranged in descending order according to the feature similarity values, a number of reference mapping features with the highest feature similarity to the first mapping feature are selected as positive samples, and negative samples are selected according to the remaining reference mapping features; The selected positive samples and negative samples are substituted into a pre-selected contrast loss function, and the contrast loss function is used to calculate the loss value based on the associated probability. The parameters are adjusted by the loss value, so as to reduce the similar image feature distance between the first mapping feature and the reference mapping feature and between the third mapping feature and the reference mapping feature, and increase the dissimilar image feature distance.

8. The unsupervised cross-view image geolocation method based on graph structure according to claim 7, characterized in that: Negative samples are selected based on the remaining reference mapping features, including: All other reference mapping features are used as negative samples, or the negative samples are determined from all other reference mapping features using a difficult sample selection method.

9. The unsupervised cross-view image geolocation method based on graph structure according to claim 7, characterized in that: The method further includes: taking the reference mapping feature that ranks first in feature similarity with the first mapping feature as a positive sample, and taking all other reference mapping features as negative samples.

10. The unsupervised cross-view image geolocation method based on graph structure according to claim 1, characterized in that: The method further comprises: Collecting a first-perspective image, a second-perspective image, and a third-perspective image, all of which take an object located in a target area as a photographing object, to form a multi-perspective image data set, wherein the third-perspective image includes geographic location information of the target area; Dividing the multi-view image dataset into a training set and a test set; Construct the initial graph structure; According to a preset goal, the initial graph structure is trained using the training set, and an optimization algorithm is selected to optimize the initial graph structure until a preset training stop condition is reached; The trained initial graph structure is tested using a test set to determine the feature graph.

Images