Unsupervised cross-view image geo-localization method based on graph structure

Abstract

The application discloses an unsupervised cross-view image geographical positioning method based on a graph structure. First, a first view image to be positioned and a second view image for reference are acquired, and then feature vectors of the first view image to be positioned and the second view image for reference are extracted and mapped to the same feature space based on the feature vectors of the second view image for reference. Next, the mapped feature vector of the first view image to be positioned is input into a feature graph for graph search, a target path with a starting point being a first mapped feature and a terminal point being a third mapped feature is determined, and a third mapped feature vector is obtained. Then, a third view image containing geographical position information is determined from the third mapped feature vector, and finally, the image geographical positioning result of the first view image to be positioned is obtained. The application has the beneficial technical effects that the multi-view image fusion is utilized, the labeling cost is reduced through unsupervised learning, the feature relationship is mined with the aid of the graph structure, the geographical positioning accuracy is improved, and the application can adapt to various scenes and has strong generalization ability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision image retrieval, and particularly relates to a graph structure based unsupervised cross-view image geolocation method. BACKGROUND

[0002] With the rapid development and wide application of remote sensing sensors, the number of earth observation images is growing rapidly. Among them, there are many types of ground images and unmanned aerial images, but most of the images lack geographic location information. At the same time, remote sensing satellite images usually contain detailed geographic location information and can provide users with location reference of ground objects. However, due to the large difference in perspective between ground images, unmanned aerial images and remote sensing satellite images, the features of the same ground object under different perspectives are significantly different, which makes it difficult to associate different perspective images. Therefore, exploring a cross-perspective image geolocation method that can quickly and effectively achieve cross-perspective image geolocation is a difficult problem to be solved in the field of computer vision image retrieval.

[0003] Remote sensing image cross-perspective geolocation can be divided into supervised cross-perspective image positioning and unsupervised cross-perspective image positioning according to whether true value labels are used. The former is to find the corresponding satellite image of the ground image based on the category label of the ground image and the satellite image, and then perform supervised training. The latter is to find the spatial correlation of different perspective images without label information, so as to associate different perspective images.

[0004] The current cross-perspective remote sensing image positioning method based on deep learning has the following two characteristics: (1) most cross-image positioning methods are based on the premise that the true value labels of different perspective images are known; (2) in the cross-positioning of ground and satellite images, generally only image transformation is performed on the two types of images for matching. However, there are a large number of unlabeled ground-satellite image pairs in actual application. If only labels are used for cross-perspective positioning, it will occupy a large amount of human resources and the positioning efficiency is not high. Only image transformation is used for ground-satellite image positioning, which will cause distortion of the original spatial features and loss of spatial information of the image in the image transformation process. Under this background, it is necessary to study a graph structure based unsupervised ground-remote sensing image cross-perspective image positioning method. SUMMARY

[0005] The technical purpose of the present application is to solve the technical problem that the existing cross-perspective image positioning technology causes distortion of the original spatial features and loss of spatial information of the image in the image transformation process, and to provide a graph structure based unsupervised cross-perspective image geolocation method.

[0006] To achieve the above technical purpose, the present application adopts the following technical scheme.

[0007] The embodiment of the present application provides a kind of unsupervised cross-view image geographic positioning method based on graph structure, comprising:

[0008] Obtain the first view image to be positioned, extract the feature vector of the original image to be positioned;

[0009] With the second view image feature vector extracted from the second view image as benchmark, the second view image feature vector and the feature vector of the original image to be positioned are mapped to the same feature space;

[0010] The mapping feature vector to be positioned after the feature vector of the original image to be positioned is mapped is input into the feature map obtained by pre-training to determine the third mapping feature vector by graph search of target path, and the starting point of the target path is the first mapping feature, and the terminal point is the third mapping feature;

[0011] Determine the corresponding third view image based on the third mapping feature vector, wherein the third view image includes image geographic location information;

[0012] Determine the geographic positioning result of the first view image to be positioned according to the image geographic location information;

[0013] Wherein, the graph node of the feature map includes the first mapping feature, the reference mapping feature and the third mapping feature in the same feature space, the edge of the feature map includes the edge between the first mapping feature and the reference mapping feature, and the edge between the reference mapping feature and the third mapping feature;Each of the edges has a corresponding weight;The first mapping feature, the reference mapping feature and the third mapping feature are determined according to the first view image, the second view image and the third view image respectively.

[0014] In some possible embodiments, the determination mode of the first mapping feature, the reference mapping feature and the third mapping feature is:

[0015] Feature extraction is respectively performed on the first view image, the second view image and the third view image to obtain the first image feature vector, the second image feature vector and the third image feature vector;

[0016] Determine the reference image feature vector according to the second image feature vector;

[0017] With the reference image feature vector as benchmark, the first image feature vector, the reference image feature vector and the third image feature vector are mapped to the same feature space, so as to obtain the first mapping feature vector, the reference mapping feature vector and the third mapping feature vector.

[0018] In some possible implementation manners, the reference image feature vector is the second image feature vector, or the reference image feature vector is a clustering center feature vector obtained by clustering the second image feature vector.

[0019] In some possible implementation manners, the reference image feature vector is a clustering center feature vector obtained by clustering the second image feature vector; each reference image feature in the reference image feature vector has a pseudo label of a corresponding cluster.

[0020] Each first mapping feature in the first mapping feature vector is assigned a pseudo label of a corresponding first mapping reference feature vector; the first mapping reference feature vector is determined as follows: for each first mapping feature, feature similarities between each reference mapping feature in the reference mapping feature vector and the first mapping feature are arranged in descending order, and a preset number of reference mapping features are selected to form the first mapping reference feature vector.

[0021] Each third mapping feature in the third mapping feature vector is assigned a pseudo label of a corresponding last mapping reference feature vector; the last mapping reference feature vector is determined as follows: for each third mapping feature, feature similarities between each reference mapping feature in the reference mapping feature vector and the third mapping feature are arranged in descending order, and the preset number of reference mapping features are selected to form the last mapping reference feature vector.

[0022] The mapped to-be-positioned mapping feature vector of the to-be-positioned original image feature vector is input into a feature map obtained by pre-training to perform graph search on a target path, and the third mapping feature vector is obtained.

[0023] The graph search is performed with the minimum sum of weights of edges in the target path and the label similarity between pseudo labels of nodes at two ends of the target path meeting a preset requirement as a target, so as to determine the third mapping feature vector.

[0024] In some possible implementation manners, the determination of the first mapping reference feature vector specifically includes:

[0025] For each first mapping feature, vector dot products of the first mapping feature and each reference mapping feature in the reference mapping feature vector are sequentially determined to obtain first feature similarities; the first feature similarities are arranged in descending order, and a preset number of reference mapping features are selected to form the first mapping reference feature vector.

[0026] Or, for each of the first mapping features, the vector dot product of each of the reference mapping features and the first mapping feature is sequentially determined to obtain each second feature similarity; each of the second feature similarities is arranged in descending order, and a preset number of reference mapping features are selected to form the first mapping reference feature vector;

[0027] Or, for each of the first mapping features, the first feature similarity and the second feature similarity are determined respectively, and both are arranged in descending order; the sizes of the first feature similarity and the second feature similarity are sequentially compared, and the reference mapping feature with the largest value is selected, and the first mapping reference feature vector is formed by selecting the first preset number of reference mapping features.

[0028] In some possible embodiments, the label similarity between the pseudo labels of the two end nodes of the target path meets a preset requirement, including:

[0029] The vector dot product operation is performed on the pseudo label of the first mapping feature vector and the pseudo label of the third mapping feature vector to obtain the label similarity;

[0030] When the label similarity is greater than or equal to a threshold value, the preset requirement is met.

[0031] In some possible embodiments, the method further includes:

[0032] For each of the first mapping features, the feature similarity between it and all reference mapping features in the reference mapping vector is calculated, and the feature similarity value is taken as the corresponding association probability;

[0033] For a single first mapping feature, the corresponding reference mapping features are arranged in descending order according to the feature similarity value, the reference mapping features with the top number of feature similarities with the first mapping feature are selected as positive samples, and negative samples are selected according to the remaining reference mapping features;

[0034] The selected positive samples and negative samples are substituted into a preselected contrast loss function, a loss value is calculated based on the association probability by using the contrast loss function, and the parameters are adjusted through the loss value, so that the similar image feature distance between the first mapping feature and the reference mapping feature and between the third mapping feature and the reference mapping feature is reduced, and the dissimilar image feature distance is increased.

[0035] In some possible embodiments, the negative samples are selected according to the remaining reference mapping features, including:

[0036] All of the remaining reference mapping features are taken as negative samples, or the negative samples are determined from all of the remaining reference mapping features by using a difficult sample selection method.

[0037] In some possible implementation manners, the method further includes: taking a reference mapping feature with a feature similarity to the first mapping feature as a positive sample, and taking all the remaining reference mapping features as negative samples.

[0038] In some possible implementation manners, the method further includes:

[0039] The first-view image, the second-view image, and the third-view image are all taken as a shooting object of an object located in a target region to form a multi-view image dataset, and the third-view image includes geographical position information of the target region.

[0040] The multi-view image dataset is divided into a training set and a test set.

[0041] An initial graph structure is constructed.

[0042] According to a preset target, the initial graph structure is trained by using the training set, and an optimization algorithm is used to optimize the initial graph structure until a preset training stop condition is reached.

[0043] The trained initial graph structure is tested by using the test set to determine the feature graph.

[0044] Compared with the prior art, the method for unsupervised cross-view image geographical positioning based on a graph structure provided in the embodiments of the present application has the beneficial technical effects that: many traditional image geographical positioning methods often need a large number of images with accurate geographical annotation information as supervision information to train a model, while the image geographical positioning method provided in the present application is unsupervised, and does not need to perform complicated geographical annotation on images in advance. In addition, by obtaining images of different views, extracting image features, mapping the image features to the same feature space, and then performing operations based on the structure of the feature graph, the hidden correlation between images of different views can be displayed in the form of a graph. When performing graph search to determine a target path, the weight settings of the nodes (first mapping features, reference mapping features, and third mapping features) and the edges in the feature graph can be used to accurately find a suitable mapping relationship according to the correlation closeness between features, so that the first-view image to be positioned is gradually associated to the third-view image containing geographical position information, thereby improving the accuracy of the final geographical positioning result and reducing the positioning error caused by the difference in view angles. BRIEF DESCRIPTION OF DRAWINGS

[0045] The drawings described herein are for purposes of illustration only and are not intended to limit the scope of the present disclosure in any way. Additionally, the shapes and relative sizes of the various components in the drawings are exaggerated for clarity and are not necessarily drawn to scale. Those skilled in the art will recognize that the various components in the drawings can be implemented using various possible shapes and relative sizes, and that the application is not limited to the shapes and relative sizes of the components shown in the drawings.

[0046] Figure 1 A flowchart of a method for unsupervised cross-view image geolocation based on a graph structure is provided for an embodiment of the present application.

[0047] Figure 2 A flowchart of a process for three-view image contrast learning is provided for an embodiment of the present application.

[0048] Figure 3 A schematic diagram of the principle of the unsupervised cross-view image geolocation process is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0049] To make the technical personnel in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0050] In the description of the present application, the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features.

[0051] In the conventional ground-satellite image positioning only through image transformation, due to the limitations of the image transformation operation (such as some geometric transformation, projection transformation, etc.), the original spatial features of the image will be distorted, the original object shape, relative position relationship and other spatial features will change, and part of the spatial information will be lost, which will affect the accuracy of subsequent positioning.

[0052] To solve the above technical problems, the present application provides an unsupervised cross-view image geolocation method based on a graph structure, comprising:

[0053] Step S1, obtaining a first-view image to be positioned;

[0054] Step S2, taking the second-view image feature vector extracted from the second-view image as a reference, mapping the second-view image feature vector and the to-be-positioned original image feature vector to the same feature space;

[0055] Step S3, inputting the mapped to-be-positioned original image feature vector into the feature map obtained by pre-training to perform graph search of a target path to determine a third mapped feature vector, wherein a starting point of the target path is the first mapped feature, and an ending point of the target path is the third mapped feature.

[0056] Step S4, determining a third-view image corresponding to the third mapped feature vector based on the third mapped feature vector, wherein the third-view image comprises image geographical position information; and determining a geographical positioning result of the to-be-positioned first-view image according to the image geographical position information.

[0057] In the embodiment, the first-view image, the second-view image and the third-view image can all be taken as a shooting object of an object located in a set region range, and the object can be the same object or different objects, and the object can be a terrain feature or a building or a street view, etc. For example, the first-view image is a ground image of A street, and the second-view image and the third-view image are respectively a drone image and a satellite image in a set range (satisfying an error requirement) near A street.

[0058] In the embodiment, the graph node of the feature map comprises the first mapped feature, the reference mapped feature and the third mapped feature in the same feature space, the edges of the feature map comprise an edge between the first mapped feature and the reference mapped feature, and an edge between the reference mapped feature and the third mapped feature; each edge has a corresponding weight; the first mapped feature, the reference mapped feature and the third mapped feature are determined according to the first-view image, the second-view image and the third-view image respectively.

[0059] As an example, the determination manner of the first mapped feature, the reference mapped feature and the third mapped feature is: performing feature extraction on the first-view image, the second-view image and the third-view image respectively to obtain a first image feature vector, a second image feature and a third image feature vector; determining a reference image feature vector according to the second image feature vector; taking the reference image feature vector as a reference to map the first image feature vector, the reference image feature vector and the third image feature vector to the same feature space, thereby obtaining the first mapped feature vector, the reference mapped feature vector and the third mapped feature vector.

[0060] The graph search in step S2 can determine the edge weight and the smallest path as the target path.

[0061] In some embodiments, the weight of each edge in the feature graph is determined according to the similarity between the graph nodes at the two ends of the edge. As an example, the Euclidean distance, Manhattan distance, or Pearson correlation coefficient of the mapping features of the graph nodes at the two ends of the edge can be calculated as the similarity between the graph nodes at the two ends of the edge.

[0062] For example, for two image feature vectors (assuming they are vector A and vector B respectively, and have the same dimension), the Euclidean distance calculation formula is: where n is the dimension of the vector, a i and b i are the i-th elements of vector A and B respectively. The smaller the distance, the more similar the two image features, and a distance threshold is usually set to determine whether they are similar.

[0063] In some embodiments, Dijkstra algorithm can also be used to search the graph on the weighted adjacency matrix to find the path with the maximum similarity, thereby realizing the matching between images of different perspectives. Dijkstra algorithm is a prior art, and will not be described here.

[0064] In specific implementations, the reference image feature vector is the second image feature vector, or the reference image feature vector is each cluster center feature vector obtained after clustering the second image feature vector. The data amount of the second perspective image feature vector is usually large and may have certain redundant information and noise interference. By clustering to extract the cluster center feature as the reference image feature, a large number of similar or strongly correlated features can be clustered into a class, and the cluster center can be used to represent the features of this class, greatly simplifying the data representation form, reducing the amount of data to be processed, and to some extent weakening the influence of abnormal features caused by noise, which is conducive to improving the efficiency and accuracy of the geographic positioning process.

[0065] In some embodiments, the reference image feature vector is each cluster center feature vector obtained after clustering the second image feature vector; each reference image feature has a pseudo label corresponding to the cluster; after mapping the first image feature vector, the reference image feature vector, and the third image feature to the same feature space, the pseudo label of each reference mapping feature in the reference mapping feature vector remains unchanged.

[0066] Each first mapping feature in the first mapping feature vector is assigned the pseudo label of the corresponding first mapping reference feature vector; the first mapping reference feature vector is determined as follows: for each first mapping feature, the feature similarity between each reference mapping feature in the reference mapping feature vector and the first mapping feature is arranged in descending order, and the first mapping reference feature vector is composed of the top set number of reference mapping features.

[0067] Each third mapping feature in the third mapping feature vector is assigned a pseudo label corresponding to the last mapping reference feature vector; the last mapping reference feature vector is determined in the following manner: for each third mapping feature, the feature similarities between each reference mapping feature in the reference mapping feature vector and the third mapping feature are arranged in descending order, and the first preset number of reference mapping features are selected to form the last mapping reference feature vector.

[0068] In some embodiments, the mapped to-be-positioned image feature vector is input into a feature map obtained by pre-training to determine a third mapping feature vector through graph search of a target path, including: performing graph search with the minimum sum of weights of edges in the target path and the label similarity between pseudo labels of nodes at both ends of the target path meeting a preset requirement as the target, so as to determine the third mapping feature vector. In some embodiments, not only the sum of weights of edges in the path is minimized, but also the nodes at both ends of the path are required to have sufficient similarity in certain feature classification or attribute.

[0069] As an example, it is assumed that the multi-label vector of the first image mapping feature vector representing the starting point of the target path, the multi-label vector of the third image mapping feature vector representing the end point of the target path, and the vector dot product operation is performed on the two to obtain the label similarity Y between the two, as follows: If , it indicates that the first mapping feature vector and the third mapping feature vector contain at least one same category, and the two are necessarily similar. P is a threshold value, which can be set to 1.

[0070] The target path determined through graph search under the double condition constraints can more accurately associate the features corresponding to different view images according to reasonable logic, so that the entire process from the to-be-positioned image to the final geographic positioning is more accurate and reliable, and the accuracy of the final geographic positioning result is improved.

[0071] In some embodiments, the weight of the edge can also be the label similarity between the pseudo labels of the nodes at both ends of the edge. The label similarity can be determined by using methods such as Euclidean distance, Manhattan distance, or Pearson correlation coefficient.

[0072] In some embodiments, as shown in Figure 2 the unsupervised cross-view image geographic positioning method based on the graph structure further includes: using a contrast learning method to reduce the similar image feature distance between the first mapping feature and the reference mapping feature and between the third mapping feature and the reference mapping feature, and increase the dissimilar image feature distance. Taking the first mapping feature as an example, it specifically includes:

[0073] For each first mapping feature, a feature similarity between the first mapping feature and all reference map features in the reference mapping vector is calculated, and the feature similarity value is taken as a corresponding association probability;

[0074] For a single first mapping feature, the corresponding reference mapping features are arranged in descending order according to the feature similarity values, the reference mapping features with the top selected number (the specific number can be determined according to a preset selected number or the like) of feature similarities with the first mapping feature are selected as positive samples, and negative samples are selected from the remaining reference mapping features.

[0075] The selected positive samples and negative samples are substituted into a preselected contrast loss function, a loss value is calculated based on the association probability by using the contrast loss function, and the parameters are adjusted through the loss value, so as to reduce the similar image feature distance between the first mapping feature and the reference mapping feature and increase the dissimilar image feature distance.

[0076] The method of reducing the similar image feature distance and increasing the dissimilar image feature distance between the third image feature and the reference image feature is the same as above, and will not be repeated here.

[0077] In some embodiments, the negative samples are selected from the remaining reference mapping features, including: all the remaining reference mapping features are taken as negative samples, or the negative samples are determined from all the remaining reference mapping features by using a difficult sample selection method.

[0078] In a specific implementation, step S3 can pre-establish a feature index library, establish an index relationship between the third mapping feature extracted from the third view image and the corresponding third view image, and construct a feature index library. The third mapping feature determined by the graph search is taken as a query condition, and a matching search is performed in the constructed feature index library. For example, a nearest neighbor search algorithm (such as a K-nearest neighbor algorithm) can be used to calculate the similarity between the third mapping feature and each stored third mapping feature in the index library (such as using a similarity measurement method such as Euclidean distance or cosine similarity to determine the similarity), find the most similar third mapping feature stored in the index library, and then determine the corresponding third view image according to the index relationship.

[0079] In a specific embodiment, the unsupervised cross-view image geolocation method based on a graph structure further includes constructing an initial graph structure first, training the graph structure to obtain a feature graph, and taking the feature graph as an example, as shown in FIG. 1, the process of training includes: Figure 3

[0080] Step S101, collect first view images, second view images and third view images, and construct a multi-angle image library, wherein the third view images include geographic location information of a target region.

[0081] ​As an example, the first-view image is a ground observation image, such as a ground-view image; the second-view image is a hollow observation image, such as a UAV-view image; and the third-view image is a high-altitude observation image, such as a satellite-view image, and the satellite image includes image geographical position information. The images of different heights have completely different imaging characteristics and features, providing more diverse feature data for the unsupervised learning process, so that the inherent relationship between images, the mapping relationship of different view features in the same feature space, and the association logic in the graph structure can be better mined based on the rich features, so that the unsupervised learning can play a more full role, and accurate geographical positioning can still be achieved while reducing the labeling cost.

[0082] The UAV-view images in the test set can be divided into a first-height UAV-view image set and a second-height UAV-view image set according to the height.

[0083] In the embodiment, the University-1652 dataset can be used, which is collected from three different platforms, namely a simulated UAV platform, satellite images, and street view images. The University-1652 dataset is divided into a training set and a test set Two sub-image libraries, wherein the training set is used for network training of the graph structure, and the test set is used for cross-view positioning result evaluation.

[0084] In the initial training stage, an equal number of images of the three views are randomly selected.

[0085] Step 102, image feature extraction is performed on the first-view image, the second-view image, and the third-view image respectively, to obtain corresponding first-view image feature vectors, second-view image feature vectors, and third-view image feature vectors.

[0086] Due to the different views, the image features of the three views are quite different, and in specific embodiments, a three-view convolutional network (three parallel branch convolutional networks) can be constructed to process the images of the three views at the same time.

[0087] In some embodiments, for the UAV-view images and the satellite-view images, since the heights are close and the shooting angles are not much different. For the images of these two views, the branch convolutional networks for these two images can share weights when designing the three-view convolutional network for processing.

[0088] As an example, a ResNet residual network can be used for image feature extraction, but the feature extraction method designed by the present application is not limited to the ResNet residual network, and other networks can also be used in specific implementation.

[0089] Step 103, clustering the second image feature vectors to obtain a cluster center feature vector as a reference image feature vector, K-means clustering (other clustering methods can also be used to realize clustering) is performed on all second-view image features in the data set, and finally a cluster center feature vector is obtained An unsupervised clustering method based on unmanned aerial vehicle view image features is prior art, and will not be described herein.

[0090] Step 104, mapping the first image feature vector, the reference image feature vector and the third image feature vector to the same feature space to obtain a first mapping feature vector, a reference mapping feature vector and a third mapping feature vector, taking the reference image feature vector (i.e. the cluster center feature vector obtained in step 104 ) as a reference.

[0091] As an example, the specific process of feature mapping includes: assuming represents the first-view image feature vector, represents the third-view image feature vector, represents the second-view image cluster center feature vector, i.e. the reference image feature vector, wherein, , represents a certain feature vector of a certain view, d0 represents the dimension of the image feature, and d is the dimension of the mapped feature. Here, a linear matrix with a size of d0xd and a parameter of θ (which can be optimized through contrast learning later) is used. The image feature vectors of different views are input into the adapter Θ, which will process the input feature vectors according to the parameters and operation rules set in itself and map them to the same feature space:

[0092] (1)

[0093] Step 105, based on the mapping feature vectors of different views in the same feature space obtained in step 104, each first mapping feature is assigned a pseudo label corresponding to the first mapping reference feature vector; the determination method of the first mapping reference feature vector is that, for each first mapping feature, the feature similarities between each reference mapping feature in the reference mapping feature vector and the first mapping feature are arranged in descending order, and the first mapping reference feature vector is composed of the first set number of reference mapping features;

[0094] Each third mapping feature is assigned a pseudo label corresponding to the last mapping reference feature; the determination method of the last mapping reference feature is that, for each third mapping feature, the feature similarities between each reference mapping feature in the reference mapping feature vector and the third mapping feature are arranged in descending order, and the last mapping reference feature vector is composed of the last set number of reference mapping features.

[0095] Taking the first-view (ground view) first-mapping feature vector and the reference mapping feature vector as an example, firstly, for a certain first-mapping feature in the first-mapping feature vector... The reference mapping feature vector of the second perspective and a certain reference mapping feature (cluster center feature of UAV view image). The two images are then subjected to a vector dot product operation, which involves determining the vector dot product between the reference image features and the first image features to obtain the first feature similarity. (As shown in Formula 2); determine the vector dot product of the first image feature and the reference image feature to obtain the second feature similarity. (As in Formula 3):

[0096] (2)

[0097] (3)

[0098] Then, assuming that the image mapping features of the two perspectives satisfy the same pseudo-label distribution, the pseudo-labels of the UAV perspective images are obtained by unsupervised clustering.

[0099] The methods for determining the first mapping reference feature vector specifically include:

[0100] For each first mapping feature, the vector dot product between the first mapping feature and each reference mapping feature in the reference mapping feature vector is determined sequentially to obtain the similarity of each first feature; the similarities of each first feature are arranged in descending order, and the first set number of reference mapping features are selected to form the first mapping reference feature vector;

[0101] Alternatively, for each first mapping feature, determine the vector dot product between each reference mapping feature and the first mapping feature in turn to obtain the similarity of each second feature; arrange the similarities of each second feature in descending order, and select the first set number of reference mapping features to form the first mapping reference feature vector;

[0102] Alternatively, for each first mapping feature, determine the similarity of each first feature and the similarity of each second feature, and arrange them in descending order; compare the magnitudes of the first feature similarity and the second feature similarity in turn, and select the reference mapping feature with the largest value, until a predetermined number of reference mapping features are selected to form the first mapping reference feature vector.

[0103] Assume the first mapping feature pseudo-label of the ground image consists of In one embodiment, Selectable or Sort the pseudo-labels corresponding to the cluster center features of the drones in descending order (a total of n); or compare... or , select the largest unmanned aerial vehicle cluster center feature in both groups, until the preset number (n) of unmanned aerial vehicle cluster center features corresponding to the pseudo label are selected. The pseudo label can be compared for label similarity when performing graph search to determine the target path.

[0104] In some embodiments, the method further comprises step 106: using a contrast learning method to narrow the distance between image features of the same location and to widen the distance between image features of different locations in the first mapping feature of the first perspective and the reference mapping feature of the second perspective. The principle can be seen from Figure 2

[0105] For each first mapping feature, the feature similarity between it and all reference mapping features is calculated, and the feature similarity value is taken as the corresponding association probability (i.e., the probability that the ground image mapping feature belongs to a certain cluster center feature in the unmanned aerial vehicle perspective image cluster feature); for a single first mapping feature, the corresponding reference mapping features are arranged in descending order according to the feature similarity value, and the reference mapping features with the top number (such as the first) similarity to the first mapping feature are selected as positive samples, and negative samples are selected according to the remaining reference mapping features (such as all the remaining reference mapping features as negative samples, or the pseudo label K-1 elements corresponding to the unmanned aerial vehicle cluster center feature can be selected as difficult negative samples using the difficult learning method); the selected positive and negative samples are substituted into the preselected contrast loss function, the loss value is calculated based on the association probability using the contrast loss function, and the parameters are adjusted through the loss value, so as to reduce the similar image feature distance between the first mapping feature and the reference mapping feature, and increase the dissimilar image feature distance.

[0106] Using the contrast learning method, the distance between image features of the same location is narrowed and the distance between image features of different locations is widened in the third mapping feature of the third perspective and the reference mapping feature of the second perspective. The method is the same as above and will not be repeated. By narrowing the feature distance between images of the same location from different perspectives (first perspective and second perspective, third perspective and second perspective), the internal relationship between features from different perspectives but representing the same actual location can be further strengthened.

[0107] For the contrast learning method, the loss function is defined based on the InfoNCE loss function as shown below:

[0108] (4)

[0109] (5)

[0110] (6)

[0111] wherein, and​ respectively represent the asymmetric hard sample contrastive learning loss function of the first view image (ground image) mapping feature vector / third view image (satellite image) mapping feature vector to the UAV cluster center feature vector (reference mapping feature vector). Ω represents the number of first view images (ground images) / the number of third view images (satellite images), τ is a temperature parameter, and controls the smoothness of the similarity distribution. and respectively represent the i-th first mapping feature and the j-th third mapping feature. and respectively represent the UAV cluster center feature closest to the i-th first mapping feature and the j-th third mapping feature as positive samples. N and M respectively represent the K UAV cluster center negative sample sets closest to the i-th first mapping feature or the j-th third mapping feature.

[0112] Step 107, construct the initial graph structure G with the final first mapping feature, the reference mapping feature (UAV view image cluster center feature), and the third mapping feature obtained in step 106. The graph G nodes include three types, namely ground image mapping feature (first mapping feature) nodes G i , UAV image mapping feature (reference mapping feature) nodes D j , and satellite image mapping feature (third mapping feature) nodes S k . Each node is associated with the feature vector obtained from the feature extraction network in the previous step. Among them, the feature vector of the ground image mapping feature node G i is f(G i ), the feature vector of the UAV image mapping feature node D j is f(D j ), and the feature vector of the satellite image mapping feature node S k is f(S k ).

[0113] Then, the similarity between the ground image mapping feature node and the UAV image mapping feature node and the similarity between the satellite image mapping feature node and the UAV image mapping feature node are calculated respectively as the edge weight.

[0114] Suppose the edge weight between the ground image mapping feature node G i and the UAV image mapping feature node D j is , and the edge weight between the satellite image mapping feature node and the UAV image mapping feature node is . The calculation of the two is as follows:

[0115] (7)

[0116] (8)

[0117] If two nodes are adjacent to each other, the edge exists, and vice versa, the edge does not exist. is a weighted adjacency matrix, which is calculated by the following formula:

[0118] (9)

[0119] According to the preset target, the initial graph structure is trained by using the training set, an optimization algorithm is selected to optimize the initial graph structure until the preset training stopping condition is reached, and the trained initial graph structure is tested by using the test set to determine the feature graph.

[0120] The method provided in the application adopts multi-view images, can integrate different scale information, realizes the consideration of details and the whole, and improves the positioning accuracy. Based on unsupervised learning, the data labeling burden is reduced, and the generalization ability is enhanced. With the help of graph structure, edge weight, node pseudo label and the like, the feature correlation is deeply mined, the positioning reliability is guaranteed, and the method is suitable for diversified geographic scenes.

[0121] The above describes in detail the unsupervised cross-view image geographic positioning method based on the graph structure provided in the application, specific examples are applied in the present application to explain the principles and implementation modes of the application, and the above description of the examples is only used to help understand the concept of the application and should not be understood as limiting the protection scope of the application.

Claims

1. An unsupervised cross-view image geo-localization method based on graph structure, characterized in that, The method comprises the following steps: Collecting a first view image, a second view image and a third view image, all of which are taken as the shooting object of an object in a positioning area, wherein the third view image comprises image geographical position information; the first view image is a ground view image, the second view image is a UAV view image, and the third view image is a satellite view image; Obtaining a first view image to be positioned and extracting a feature vector of the original image to be positioned; Taking a second view image feature vector extracted from the second view image as a reference, mapping the second view image feature vector and the feature vector of the original image to be positioned to the same feature space; Inputting the mapped feature vector of the original image to be positioned into a feature map obtained by pre-training to determine a third mapping feature vector through graph search of a target path, wherein the starting point of the target path is a first mapping feature, and the terminal point is a third mapping feature; Based on the third mapping feature vector, determining a corresponding third view image; According to the image geographical position information, determining a geographical positioning result of the first view image to be positioned; The graph nodes of the feature map comprise a first mapping feature, a reference mapping feature and a third mapping feature in the same feature space, and the edges of the feature map comprise edges between the first mapping feature and the reference mapping feature and edges between the reference mapping feature and the third mapping feature; Each edge has a corresponding weight; the first mapping feature, the reference mapping feature and the third mapping feature are respectively determined according to the first view image, the second view image and the third view image.

2. The graph structure based unsupervised cross-view image geo-localization method according to claim 1, wherein, The determination manner of the first mapping feature, the reference mapping feature and the third mapping feature is as follows: Respectively extracting features from the first view image, the second view image and the third view image to obtain a first image feature vector, a second image feature vector and a third image feature vector; Determining a reference image feature vector according to the second image feature vector; Mapping the first image feature vector, the reference image feature vector and the third image feature vector to the same feature space according to the reference image feature vector, thereby obtaining the first mapping feature vector, the reference mapping feature vector and the third mapping feature vector.

3. The graph structure based unsupervised cross-view image geo-localization method according to claim 2, wherein, The reference image feature vector is the second image feature vector, or the reference image feature vector is each cluster center feature vector obtained by clustering the second image feature vector.

4. The graph structure based unsupervised cross-view image geo-localization method according to claim 2, wherein, The reference image feature vector is each cluster center feature vector obtained by clustering the second image feature vector; each reference image feature in the reference image feature vector has a pseudo label of the corresponding cluster; Each first mapping feature in the first mapping feature vector is assigned a pseudo label of a corresponding first mapping reference feature vector; the determination manner of the first mapping reference feature vector is as follows: for each first mapping feature, arranging the feature similarity between each reference mapping feature in the reference mapping feature vector and the first mapping feature in descending order, and selecting a preset number of reference mapping features to form the first mapping reference feature vector. Each of the third mapping features is assigned a pseudo label of a corresponding last mapping reference feature vector; the last mapping reference feature vector is determined in the following manner: for each of the third mapping features, feature similarities between each reference mapping feature in the reference mapping feature vector and the third mapping feature are arranged in descending order, and the first preset number of reference mapping features are selected to form the last mapping reference feature vector; The mapping feature vector after the mapping of the to-be-positioned original image feature vector is input into a feature map obtained by pre-training to perform graph search on a target path, and a third mapping feature vector is obtained, including: The graph search is performed with the minimum weighted sum of edges in the target path and the label similarity between pseudo labels of nodes at two ends of the target path satisfying a preset requirement as a target, so as to determine the third mapping feature vector.

5. The graph structure based unsupervised cross-view image geo-localization method according to claim 4, characterized in that, The determination manner of the first mapping reference feature vector specifically includes: For each of the first mapping features, vector dot products of the first mapping feature and each reference mapping feature in the reference mapping feature vector are sequentially determined to obtain first feature similarities; the first feature similarities are arranged in descending order, and the first preset number of reference mapping features are selected to form the first mapping reference feature vector; Or, for each of the first mapping features, vector dot products of each reference mapping feature and the first mapping feature are sequentially determined to obtain second feature similarities; the second feature similarities are arranged in descending order, and the first preset number of reference mapping features are selected to form the first mapping reference feature vector; Or, for each of the first mapping features, the first feature similarities and the second feature similarities are respectively determined and are both arranged in descending order; the first feature similarity and the second feature similarity are sequentially compared, and the reference mapping feature with the maximum value is selected, and the first preset number of reference mapping features are selected to form the first mapping reference feature vector.

6. The graph structure based unsupervised cross-view image geo-location method according to claim 4, characterized in that, The label similarity between the pseudo labels of the nodes at two ends of the target path satisfies a preset requirement, including: The vector dot product operation is performed on the pseudo label of the first mapping feature and the pseudo label of the third mapping feature to obtain the label similarity; When the label similarity is greater than or equal to a threshold value, the preset requirement is satisfied.

7. The graph structure based unsupervised cross-view image geo-location method according to claim 1, characterized in that, The method further includes: For each of the first mapping features, a feature similarity between the first mapping feature and all reference mapping features in the reference mapping vector is calculated, and the feature similarity value is taken as a corresponding association probability; For a single first mapping feature, the corresponding reference mapping features are arranged in descending order according to the feature similarity values, the reference mapping features with the feature similarities with the first mapping feature arranged in the front row are selected as positive samples, and negative samples are selected according to the remaining reference mapping features; The selected positive sample and negative sample are substituted into the preselected contrast loss function, a loss value is calculated based on the correlation probability by using the contrast loss function, and the parameter is adjusted through the loss value, so as to reduce the similar image feature distance between the first mapping feature and the reference mapping feature and between the third image mapping feature and the reference mapping feature, and increase the dissimilar image feature distance.

8. The graph structure based unsupervised cross-view image geo-location method according to claim 7, characterized in that, The negative sample is selected according to the remaining reference mapping features, including: All the remaining reference mapping features are used as negative samples, or the difficult sample selection method is used to determine the negative samples from all the remaining reference mapping features.

9. The graph structure based unsupervised cross-view image geo-location method according to claim 7, characterized in that, The method further includes: taking the reference mapping feature with the first mapping feature similarity as the positive sample, and taking all the remaining reference mapping features as the negative samples.

10. The graph structure based unsupervised cross-view image geo-localization method according to claim 1, wherein, The method further includes: The first view image, the second view image and the third view image are collected, and a multi-view image dataset is formed, wherein the object in the target region is taken as the shooting object, and the third view image includes geographical position information of the target region; The multi-view image dataset is divided into a training set and a test set; An initial graph structure is constructed; According to a preset target, the initial graph structure is trained by using the training set, and an optimization algorithm is selected to optimize the initial graph structure until a preset training stop condition is reached; The trained initial graph structure is tested by using the test set, and the feature map is determined.

Images