Cross-view geolocalization method with global relational attention guidance

Abstract

The global relationship attention guided cross-view geo-location method belongs to the technical field of computers, uses a deep residual network as a backbone network, and uses a global relationship attention module to capture more robust image global structure information for matching. A double-branch network including a global branch and a local branch is designed to capture deep features with rich semantic information and local features with multi-scale context information, respectively. In the local branch, dilated convolution is used to increase the receptive field of the feature map, and a square ring division strategy is used to divide the feature map at four scales. For the feature maps of each branch, they are converted into column vector descriptors, and then a classifier is used to obtain the predicted category of each column vector. Finally, a cross-entropy loss function is used to measure the difference between the predicted category and the real category of the image, thereby improving the geo-location accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computers, and particularly relates to a cross-view geo-localization method guided by global relationship attention. BACKGROUND

[0002] Cross-view geo-localization can be regarded as an image content-based retrieval task [1][2] , which refers to matching a query image from one platform with images from other platform databases to find images of the same geographical location. Previous studies have mainly focused on matching between ground view and satellite view, aerial view images. In recent years, with the wide application of unmanned aerial vehicles [3] , unmanned aerial vehicle view images have been added to cross-view geo-localization, and geo-localization based on unmanned aerial vehicle view and satellite images has become a current research hotspot.

[0003] With the wide application of convolutional neural networks (CNN) in image classification [4][5] , target detection [6][7] , semantic segmentation [8][9] , action recognition

[10]

[11] and other visual fields, some researchers have applied convolutional neural networks to cross-view geo-localization

[12] tasks and have made significant progress. However, most cross-view geo-localization methods mainly consider the high-level semantic information of target images, ignoring the important role of spatial structure information in effectively improving geo-localization accuracy. Zheng et al.

[13] treated geo-localization as a classification task and performed similarity measurement on image semantic features. However, this method ignores the context information of the region around the target, resulting in insufficient comprehensive features. Wang et al.

[14] divided image high-level features according to a square ring division strategy, performed similarity measurement on each part of the features, and thus improved geo-localization accuracy by using context information. However, this method directly divides the feature map into four scales, ignoring the global structure information of the image, resulting in the situation that similar images are regarded as correct retrieval results in the retrieval process, causing false detection. Obviously, fully mining the structure information of the geographical target image is helpful to improve the performance of cross-view geo-localization. SUMMARY

[0004] In view of the fact that most algorithms fail to fully consider the influence of image structure information on matching accuracy in cross-view geo-localization, the application provides a cross-view geo-localization method guided by global relationship attention.

[0005] First, a deep residual network

[15] is used as a backbone network, and a global relationship attention module

[16] More robust global structure information of images is captured for matching. Secondly, a dual-branch network including global branch and local branch is designed to capture deep features with rich semantic information and local features with multi-scale context information respectively. In the local branch, dilated convolution

[17] is used to increase the receptive field of feature maps, and square ring partition strategy is used to divide feature maps at 4 scales.

[14] For feature maps of each branch, the feature maps of each branch are converted into column vector descriptors, and then the predicted class of each column vector is obtained by using the classifier. Finally, the cross-entropy loss function

[18]

[19] is used to measure the difference between the predicted class of the image and the real class, so as to improve the training accuracy of the network.

[0006] The advantages are:

[0007] 1. A global relationship attention guided cross-view geolocation method is proposed, which uses a global relationship attention module to learn the relationship between image feature nodes, fully excavates the global structure information of the image, and makes the network pay attention to the salient region, so as to extract more robust features for image feature matching.

[0008] 2. A dual-branch structure including global branch and local branch is designed for cross-view geolocation. In the global branch, deep residual network is used to extract deep features to obtain feature maps containing rich semantic information; in the local branch, dilated convolution is used to capture more rich multi-scale context information.

[0009] 3. The experimental results on three data sets of University-1652, CVUSA and CVACT show that the method proposed in the technical solution obtains better performance than other advanced models in geolocation, which proves the effectiveness of the proposed method. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is the overall framework diagram of the method network of the technical solution.

[0011] Figure 2 is Figure 1 the local enlarged view of the middle part on the left.

[0012] Figure 3 is Figure 1 the local enlarged view of the middle part on the right.

[0013] Figure 4 is the global relationship attention structure diagram.

[0014] Figure 5 is the global spatial relationship attention structure diagram.

[0015] Figure 6 is a global channel relation attention structure diagram.

[0016] Figure 7 is a standard convolution and dilated convolution schematic diagram.

[0017] Figure 8 is a retrieval result for a UAV view target positioning task.

[0018] Figure 9 is a retrieval result for a UAV navigation task.

[0019] Figure 10 is a retrieval result on the dataset CVUSA. DETAILED DESCRIPTION

[0020] The related work of cross-view geo-localization is introduced here; the method and network structure used in this technical solution are described in detail; the experimental results are analyzed, and ablation experiments are carried out and summarized.

[0021] 1. Related work

[0022] The early research content of cross-view geo-localization is mainly based on ground view and aerial view images. Workman et al.

[20] Two public pre-trained models are used to extract features from images, which proves that deep features have the ability to distinguish images of different geographical locations. However, this method only focuses on single-scale image feature extraction and fails to effectively utilize multi-scale information, resulting in insufficient matching features extracted by the network. On this basis, Workman et al.

[21] constructed the CVUSA (Cross-View USA) dataset to perform multi-scale fusion on aerial image features, improving the cross-view localization results. Lin et al.

[22] used public data to construct 78000 pairs of street view and 45° aerial view images, and then used a deep twin network to extract features for cross-view localization. Vo et al.

[23] evaluated different deep learning methods, used a distance-based logistic loss layer (Distance Based Logistic Layer, DBL) and rotation invariance to train the network, and improved the localization accuracy. Considering that image semantic information is less affected by viewpoint changes, Tian et al.

[24] extracted buildings in images to perform cross-view matching and obtain the final geo-localization results. Altwaijry et al.

[25] focused on aerial image pair matching tasks, and used a data-driven method to learn discriminative representations from image pairs, solving the super-wide baseline image matching problem. In addition, Zhai et al.

[26] First, aerial image features are extracted, then an adaptive transformation is used to map aerial image features to ground view, finally, the cross-view geo-localization is completed by minimizing the difference between predicted ground view semantic features and semantic features directly extracted from ground images using an end-to-end learning method. Hu et al.

[27] The twin network is combined with NetVLAD

[28] to encode local features, obtain global image descriptors, and introduce a weighted soft-margin ranking loss to accelerate network convergence, thereby improving network performance. Shi et al.

[29] argue that existing methods ignore the differences in appearance and geometry between ground view and aerial view images, so they use polar coordinate transformation to approximately align aerial images with ground images. To further address the cross-view orientation alignment problem, Shi et al.

[30] design a dynamic similarity matching network (DSM) to align the directions of cross-view images, making the image matching result more accurate. Liu et al.

[31] argue that geometric cues can be used for localization, so they design a twin network to encode the direction information of each pixel in the image into the network model, allowing the network to simultaneously learn appearance and geometric information, improving the recall rate and precision of the network. To address the problem of scene changes over time, Rodrigues et al.

[32] propose a semantic-driven data augmentation technique to simulate the scene change phenomenon in cross-view image matching, then use a multi-scale attention module for image matching, improving network performance. Regmi et al.

[33] first apply generative adversarial networks (GANs) in cross-view localization

[34] , they use generative adversarial networks (GANs) to synthesize aerial images from ground views for image matching, but this method is not an end-to-end method. Toker et al.

[35] use polar coordinate transformation to synthesize ground views from satellite views, then perform image retrieval, integrating the two steps into an end-to-end architecture to achieve advanced geo-localization performance. The above methods mainly focus on the matching task between ground view and aerial view images, only considering two views for geo-localization, without considering the important role of unmanned aerial vehicle view images, thus lacking feature learning for multi-view matching tasks.

[0023] Recent cross-view geo-localization research believes that increasing the number of viewpoints can improve the accuracy of geo-localization, so unmanned aerial vehicles are used as the third platform to solve the problem of geo-localization. Zheng et al.

[13] The University-1652 dataset is constructed, including satellite view images, ground view images and UAV view images, all view images of the same location are regarded as a category, and the geolocation task is completed in a classification manner, and instance loss is applied to optimize the model

[36] . However, this method only focuses on the semantic information of the image and does not consider the influence of the detailed information of the image on cross-view geolocation. To solve this problem, Wang et al.

[14] proposed a local pattern network (LPN) that uses the context information of the image as an auxiliary clue and uses a square ring division strategy to make the network pay attention to the environment around the target building, effectively solving the problem of ignoring image detail information in method

[13] , and achieving good matching results. Ding et al.

[37] proposed a cross-matching method based on location classification (LCM) to solve the problem of unbalanced satellite and UAV image samples and improve image matching accuracy. Attention mechanism has been widely used in computer vision

[16]

[38]

[39]

[40] , which aims to make the network pay more attention to the distinguishing features and filter out some irrelevant information to improve the training effect of the model. Zhang et al.

[16] integrated global relationship attention into the pedestrian re-identification network, which improved the feature representation ability by capturing the global structure information of the image and improved the performance of pedestrian re-identification. To avoid the influence of target shift and view scaling on image matching, Zhuang et al.

[38] proposed a multiscale block attention (MSBA) structure to strengthen the salient features of different regions of the feature map. Lin et al.

[39] designed a unit subtraction attention module (USAM) to detect key points in the feature map, making the model pay attention to the salient regions of the image, and improving the performance of the model with fewer parameters. Dai et al.

[40] believe that some operations based on convolutional neural networks (CNN) will cause the loss of fine-grained information of the image, so they introduced the Transformer structure

[41] into cross-view positioning and proposed a feature segmentation and region alignment method (FSRA) that divides the feature map into different regions according to the heat distribution and classifies each region, effectively realizing cross-view positioning.

[0024] The above method provides a new research idea for solving the problem of inaccurate cross-view geolocation. Inspired by this, the method fully combines attention mechanism with feature extraction network to mine structural information from a global perspective. At the same time, a double-branch structure is adopted to jointly train global and local image features, and dilated convolution is fused in the local branch to increase the receptive field of the feature map, so as to capture more rich multi-scale context information and improve the accuracy of cross-view positioning.

[0025] Cross-view geolocation is an important research direction in the field of image retrieval, mainly used to match images from different platforms of the same geographic location. Most existing methods fail to fully consider the role of image structure information in cross-view geolocation, resulting in features that cannot fully represent images, which affects the positioning accuracy. Based on this, this paper proposes a global relationship attention guided cross-view geolocation method, which fully integrates global relationship attention with feature extraction network, enabling the network to capture rich global structural information and improve the representation ability of features. At the same time, considering the important role of semantic information and context information in geolocation, a joint training structure of global branch and local branch in parallel is designed to fully mine multi-scale context features for image matching, further improving the accuracy of cross-view geolocation. Quantitative and qualitative experimental results on datasets University-1652, CVUSA and CVACT show that the algorithm of the technical solution has significant advantages in recall and image retrieval accuracy (AP) indicators.

[0026] 2、Method

[0027] 2.1 Network framework

[0028] The network framework proposed by the technical solution is as follows Figure 1As shown, the whole network structure is divided into global branch and local branch, which share network weights. Firstly, the technical solution adopts ResNet50 as the backbone network, removes the average pooling layer and classification layer, and then extracts the image features of the input. At the same time, after extracting the shallow layer features, the global relation attention module (Relation-Aware Global Attention, RGA) is added, including global spatial relation attention (Spatial Relation-Aware Global Attention, RGA-S) and global channel relation attention (Channel Relation-Aware Global Attention, RGA-C), which is used to capture the global structure information of the image. Secondly, the double branch structure is used to process the output features of the previous stage, so as to effectively pay attention to the global and local information. The global branch is used to extract the high-level semantic information of the whole image; the local branch is used to focus on the deep features of the network, so as to retain more image detail information. At the same time, in order to combine the information of the surrounding area of the target, the feature map is divided into four different regions using the square ring division strategy in the local branch. Finally, the global average pooling is used to convert the image high-level features into column vector descriptors. In the training stage, the classifier module is used to obtain the predicted class probability of each column vector descriptor, and the cross entropy loss function is used to minimize the difference between the predicted class and the real class. The Euclidean distance is used to calculate the similarity between the query image and the database image in the test process, and finally the retrieval results are sorted based on the similarity.

[0029] 2.2 Global Relation Attention Module

[0030] In the cross-view geolocation task, the global relation attention module can make the network pay attention to the different features in the image, helping to distinguish buildings with similar appearances. The global relation attention module is combined with the feature extraction network, including global spatial relation attention (RGA-S) and global channel relation attention (RGA-C), to construct a global relation attention guided feature extraction network, which calculates the attention weight by learning the relationship between feature nodes. As shown in Figure 4 , for the feature vector in the feature map, the feature vector is represented as a feature node x i , i = 1, 2, … N, N is the number of feature nodes. For a certain feature node x i , the correlation r i,j between the current node and other nodes is calculated j,i , j = 1, 2, … N, so that the relation vector of the feature node x i is r i = [r i,1 , r i,2,…,r i,N ,r 1,i ,r 2,i ,…,r N,i Then, the feature nodes are compared with the relation vector r. i By concatenating these features, we obtain relation-aware features, which are then used to infer the attention weights of the current feature nodes.

[0031] 2.2.1 Global Spatial Relationship Attention

[0032] Global Spatial Relationship Attention (RGA-S) learns the correlations between all feature nodes in the spatial dimension of the feature map, enabling the network to capture features of salient target regions. Global Spatial Relationship Attention, for example... Figure 5 As shown:

[0033] Specifically, for feature maps obtained from neural networks Each spatial location's C-dimensional feature vector is treated as a feature node, forming a graph G with a total of N = W × H nodes. S The feature node is represented as x i Where i = 1, 2, ..., N. Feature node x i With x j The correlation between them r i,j It can be obtained through the dot product operation, specifically defined as equation (1):

[0034] r i,j =f s (x i ,x j )=(ReLU(BN(Conv(x i )))) T (ReLU(BN(Conv(x j )))) (1)

[0036] Where f s (·) represents the dot product operation, ReLU(·) represents the modified linear unit activation function, BN(·) represents the batch normalization layer, and Conv(·) represents the 1×1 convolution operation. The dimensionality reduction ratio is controlled by a predefined positive integer. Similarly, the feature node x can be obtained. j With x i The correlation between them r j,i , using (r i,j ,r j,i To represent feature node x i With x j The pairwise relationships between them. Finally, a relation matrix is ​​used. Represents the correlation between all nodes, where r i,j =RS (i,j).

[0037] Stacking the correlation of the i-th feature node with all nodes in a fixed order to obtain a spatial relationship vector where R S (i,:) represents the correlation between the i-th feature node and all nodes, and R S (:,i) represents the correlation between all nodes and the i-th node. In order to enable the network to fully utilize the global structure information of the feature nodes, the spatial relationship vector r i is spliced with the feature node x i itself to obtain the spatial relationship-aware feature E S , which can be defined as formula (2):

[0038] E S =C(x i ,r i ) = (pool c (ReLU(BN(Conv(x i )))), (ReLU(BN(Conv(r i )))))

[0039] (2) where C(·) represents the splicing operation, and pool c (·) represents global average pooling in the channel dimension, reducing the channel dimension to 1. The spatial attention weight a i of the i-th feature can be calculated through the spatial relationship-aware feature, which is defined as formula (3):

[0040] a i = sigmoid(BN(Conv2(ReLU(BN(Conv1(E S )))))) (3)

[0041] where sigmoid(·) represents the sigmoid activation function, Conv2(·) converts the channel number to 1, and Conv1(·) reduces the dimension at a fixed rate.

[0042] 2.2.2 Global Channel Relationship Attention

[0043] Global channel relationship attention (RGA-C) learns the relationship between all feature nodes in the channel dimension of the feature map, and gives different weights to each channel. The global channel relationship attention is as shown in Figure 6 :

[0044] Specifically, for a feature map , the feature map on each channel is taken as a feature node to form a graph G CEach feature node is represented as x i , where i = 1, 2, ..., C.

[0045] For the input feature map S, spatially compress the input feature map S into... Feature node x can be obtained i With feature node x j The correlation between them r i,j Defined as Equation (4):

[0046] r i,j =f c (x i ,x j )=(ReLU(BN(Conv(x i )))) T (ReLU(BN(Conv(x j (4)

[0047] Where f c (·) represents the dot product operation. Similarly, we obtain the feature node x. j With x i The correlation between them r j,i Using matrices This represents the correlation between all nodes. The channel relationship vector is obtained by stacking the correlations between the i-th feature node and all nodes. Similar to equations (2) and (3), the channel relationship characteristic E can be obtained. C and channel attention weight a i .

[0048] 2.3 Local Branches

[0049] To enable the network to capture rich multi-scale contextual information and preserve more detailed spatial structure information, thereby facilitating the retrieval of the most similar images in the database and improving the accuracy of cross-view geolocation, dilated convolutions with various dilation rates are utilized in local branches.

[17] This approach increases the receptive field of the feature map without sacrificing image details, thereby improving the network's ability to capture multi-scale information. Simultaneously, a square ring partitioning strategy is employed to segment the feature map across four scales, obtaining rich spatial contextual information.

[0050] Dilated convolution expands the receptive field by inserting r-1 weights of 0, where r is the dilation factor, which is 1 in standard convolution operations. For example... Figure 7 The diagram shows the standard convolution and dilated convolution structures, where... Figure 7 Part (a) represents standard convolution. Figure 7The middle part (b) represents the dilated convolution with a dilation factor of 2. Using a 3x3 kernel under the same conditions, the receptive fields of the standard convolution and the dilated convolution are 3x3 and 5x5, respectively. Compared with the standard convolution, the dilated convolution can capture more rich multi-scale information of the image for image matching.

[0051] Specifically, the module uses dilated convolution operations with dilation factors of 2 and 4, respectively, to increase the receptive field of the feature map. At the same time, the step length of the convolution layer and the down-sampling layer in the last residual block of the ResNet50 is adjusted to 1. When the resolution of the input image is 256x256, the resolution of the feature image output by the backbone network is 8x8, and the resolution of the feature image output by the dilated residual network is 32x32.

[0052] In order to help the network better distinguish images in different geographical locations, the environment around the target building is taken as auxiliary information, and a square ring division strategy is used in the local branch to divide the feature image into four parts according to the distance to the center of the image, and the feature maps of different regions are obtained. Then, the image features are converted into 2048-dimensional feature vectors through average pooling operation, which is represented by formula (5):

[0053]

[0054] where Avgpool(·) represents the average pooling operation, represents the divided local branch feature map in different view platforms, represents the 2048-dimensional feature vector after pooling of the four local branches.

[0055] 2.4 Global Branch

[0056] Considering that the semantic information concerned by the deep network is also an important part of the cross-view geolocation task, a global branch structure parallel to the local branch is designed. In the global branch, a deep residual network is used to extract and refine large-scale features, thereby obtaining a feature map f j , so that the network can identify the category to which different image features belong. Then, the global feature map is processed by average pooling to obtain a 2048-dimensional feature vector, which is represented by formula (6):

[0057] g j = Avgpool(f j ) (6)

[0058] where g j represents the global branch feature vector after pooling.

[0059] 2.5 Classification Learning and Loss Function

[0060] The technical solution fuses the classifier into the feature extraction stage, and uses the classifier to predict the category of each feature vector. The classifier is composed of a fully connected layer (Fully Connected Layer, FC), a batch normalization layer (Batch Normalization Layer, BN), a Dropout layer (Dropout Layer) and a classification layer (Classification Layer, Cls). The local feature vector and the global feature vector g j are input to predict the category to which each feature vector belongs, and finally obtain the local prediction probability distribution vector and the global prediction probability distribution vector q j .

[0061] The loss function used in the technical solution is cross-entropy loss, which is used to measure the distribution difference between the predicted probability and the true probability of the image, so as to better learn the image features and improve the network training accuracy. The cross-entropy loss can be represented by formula (7):

[0062]

[0063] wherein represents the corresponding original image after processing by the ring division strategy, x j (j∈[1,2]) represents the input image, j=1 represents the unmanned aerial vehicle platform, and j=2 represents the satellite platform. y represents the true category of the input image, respectively represent and x j , and the normalized probability score of the true category is defined by formula (8) and formula (9),

[0064]

[0065]

[0066] wherein C represents the number of all geographic label categories in the database.

[0067] 3 Experiment

[0068] 3.1 Dataset

[0069] The technical solution uses University-1652

[13] , CVUSA

[21] and CVACT

[31] three data sets to train and test the proposed method.

[0070] (1) University-1652 is a multi-view multi-source dataset, which includes 1652 drone, satellite and ground view images of 72 universities, and there are no repeated images between the training dataset and the test dataset. We use the drone and satellite images in this dataset to study two tasks: drone view object localization and drone navigation. In the drone view object localization task, there are 701 image classes in the query image database of the drone view, and each class corresponds to a real matching satellite image. In the drone navigation task, there are 701 image classes in the query database of the satellite view, and each class corresponds to 54 real matching drone images.

[0071] (2) The CVUSA dataset includes satellite images and panoramic ground images, of which there are 35532 image pairs for training and 8884 image pairs for testing.

[0072] (3) CVACT is a larger benchmark dataset, which also includes 35532 training image pairs, but the difference is that 8884 image pairs are used as a validation dataset, and an additional 92802 image pairs are used as a test set.

[0073] 3.2 Experimental details

[0074] In order to ensure the fairness of the experiment, all algorithms are implemented on a Linux server with Ubuntu 20.04 operating system, and all performance comparisons are based on the results under this configuration. The server configuration is: GTX 3090 GPU, with a memory capacity of 24G. The model proposed in this technical solution is implemented based on the Pytorch framework. Before training, the size of all input images will be adjusted to 256x256, and horizontal flipping and random rotation are used for data augmentation. The SGD optimizer with momentum 0.9 and weight decay 0.0005 is used to update the model, and the initial learning rate is set to 0.001. In order to make the network better converge, for the dataset University-1652, the model training period is 140, and for the datasets CVUSA and CVACT, the model training period is 100. In the test phase, the similarity of images is evaluated by calculating the Euclidean distance between the query image and the database image.

[0075] 3.3 Performance comparison

[0076] 3.3.1 Quantitative comparison

[0077] The technical solution adopts recall rate (Recall@K) and image retrieval accuracy (average precision, AP) as image retrieval performance measurement indexes. Recall@K refers to the ratio of correct matching images in the first k results of retrieval to all correct images in the database, and is used to measure the recall rate. In the technical solution, the case of k = 1 is mainly considered. AP is the area under the precision-recall (PR) curve, and refers to the ratio of the retrieved true matching images to the total number of retrieval results, and is used to measure the precision rate. The greater the Recall@K and AP values, the higher the accuracy of image retrieval.

[0078] To illustrate the effectiveness of the proposed method, the proposed method is compared with other CNN-based algorithms on three datasets of University-1652, CVUSA and CVACT. The comparison results on the University-1652 dataset are shown in Table 1. The compared methods include Instance Loss

[13] , LCM

[37] , LPN

[14] , Instance Loss+USAM

[39] , LPN+USAM

[39] method.

[0079] Table 1 Quantitative test results on the dataset University-1652.

[0080] Among them, the method Contrastive Loss

[22] , Triplet Loss

[42] , Soft Margin Triplet Loss

[27] is obtained by replacing the loss function based on the method Instance Loss

[13] . The optimal and suboptimal results of the evaluation indexes are indicated by parentheses.

[0081]

[0082] As can be seen from Table 1, the method of the technical solution achieves the optimal results in both tasks of the University-1652 dataset. In the unmanned aerial vehicle view target positioning task, the algorithm achieves a performance of 81.06% and 83.74% in Recall@K and AP indexes, respectively, which is 3.99% and 3.65% higher than the suboptimal method LPN+USAM

[39] in both indexes. In the unmanned aerial vehicle navigation task, the performance in Recall@K and AP indexes is 89.58% and 79.63%, respectively. Compared with the suboptimal method LPN

[14] Recall@K and AP are improved by 3.13% and 4.84% respectively, which proves that the proposed method has significant advantages in image retrieval performance.

[0083] The comparison results on the datasets CVUSA and CVACT are shown in Table 2. Since the ground truth images in these two datasets are panoramic images, the sequential partition strategy is adopted to divide the images. The compared methods include CVM-Net

[14] ,

[27] Instance Loss

[13] , Regmi et al.

[33] , Siam-FCANet

[43] , CVFT

[12] , LPN

[14] , Instance Loss+USAM

[39] , LPN+USAM

[39] . Among them, the LPN

[14] , LPN+USAM

[39] algorithms are the results generated by training using the publicly released code, while the results of other methods are directly used by the authors.

[0084] Table 2 Quantitative test results on the datasets CVUSA and CVACT.

[0085] The optimal and suboptimal results of evaluation indicators are indicated by parentheses.

[0086] indicates that the method uses additional directional information as input.

[0087]

[0088] As can be seen from Table 2, on the CVUSA dataset, the proposed method achieves 88.00% and 99.47% in the evaluation indicators R@1 and R@Top1%, respectively. Compared with the other nine advanced models, the proposed method has significant improvements in both indicators, especially in the R@1 indicator, with a performance improvement of up to 2.03%. On the CVACT_val dataset, the proposed method achieves 80.98% and 96.53% in R@1 and R@Top1%, respectively, both of which achieve the optimal results, thereby proving the effectiveness of the proposed method.

[0089] 3.3.2 Qualitative results

[0090] Figure 8 and Figure 9 ​are the retrieval results of the proposed method on University-1652 dataset, respectively, visualizing the UAV view target localization task and the UAV navigation task, Figure 10 are the retrieval results on CVUSA dataset. In the qualitative results, each row represents the retrieval results of one location, the image on the left of the dashed line is the query image, and the image on the right is the top-ranked image in the matching results. The yellow box represents the correct retrieval, and the blue box represents the incorrect retrieval.

[0091] For the UAV view target localization, Figure 8 only one of the top five images in the matching results is the truly matched image, which proves that the proposed method can correctly retrieve the matched image under the interference of similar images. For the UAV navigation task, Figure 9 all the top five images in the matching results are the truly matched images, because each satellite image has 54 matched UAV images. In the CVUSA dataset, each ground image corresponds to a correct satellite image, Figure 10 the first image in the retrieval results of each query image is the correct matched image. Through the analysis of the qualitative results, it can be found that the proposed method can retrieve the correct results on both datasets.

[0092] 3.4 Ablation experiments

[0093] To verify the effectiveness of each module, the proposed method is analyzed in this section, and ablation experiments are conducted on the University-1652 dataset to verify the role of each module.

[0094] 3.4.1 Effectiveness of global relationship attention module

[0095] To verify the effectiveness of the global relationship attention module, two ablation experiments are conducted in this paper. The first experiment removes the global relationship attention module and only uses the double-branch structure network to extract image features. The second experiment adds the SE attention module in the network to obtain the attention of the image in the channel dimension.

[44]

[0096] According to the results in Table 3, it can be observed that compared with the results without adding attention mechanism and using the SE attention module, using the global relationship attention module can make the network focus on the discriminative features of the image, improve the retrieval ability of the network, and make the indicators achieve better results.

[0097] Table 3 Comparison of global relationship attention module ablation experiment results.

[0098] ​(a) means no attention mechanism is used; (b) means SE attention module is added; (c) means RGA attention module is used.

[0099] The optimal evaluation index is indicated by parentheses.

[0100]

[0101] 3.4.2 Effectiveness of dilated convolution

[0102] In order to verify the effectiveness of dilated convolution, three ablation experiments were conducted to adjust the dilated convolution in the local branch and use different dilation rates to extract image features. According to the results in Table 4, it can be found that using dilated convolution to increase the receptive field of the feature map can effectively capture the detailed information of the image and improve the accuracy of cross-view geolocation. When the dilation factor is 2 and 4 respectively, the performance of the model reaches the optimal.

[0103] Table 4 Comparison of dilated convolution ablation experiment results.

[0104] (a) means the dilation factor of the local branch residual block is 1, 1 respectively;

[0105] (b) means the dilation factor of the local branch residual block is 1, 2 respectively;

[0106] (c) means the dilation factor of the local branch residual block is 2, 2 respectively;

[0107] (d) means the dilation factor of the local branch residual block is 2, 4 respectively.

[0108] The optimal evaluation index is indicated by parentheses.

[0109]

[0110] 3.4.3 Effect of input image size on results

[0111] Using high-resolution images to train the model can obtain higher accuracy, but it requires more computing resources and time. Due to limited resources, low-resolution input images need to be used in actual operation, which will reduce the accuracy of image matching. Therefore, a set of ablation experiments are designed to observe the influence of input images of different resolutions on the performance of the model. The results are shown in Table 5, which shows that when the input image size increases from 224 to 320, the Recall@1 and AP (image retrieval accuracy) values of the network are improved, and when the image size increases to 384, the performance of the network slightly decreases.

[0112] Table 5 Effect of input images of different resolutions on results.

[0113] (a) means the input image size is 224x224;

[0114] (b) indicates that the input image size is 256x256;

[0115] (c) indicates that the input image size is 320x320;

[0116] (d) indicates that the input image size is 384x384.

[0117] The optimal evaluation index is indicated by parentheses.

[0118]

[0119] 4 Conclusion

[0120] The technical solution proposes a cross-view geographic positioning method guided by global relationship attention. This method uses global relationship attention to capture the global structural information of the image, and extracts more robust image features for geographic positioning. At the same time, a double-branch strategy is used for joint training, and dilated convolution is used in the local branch to increase the receptive field of the feature map, and the feature map is divided into 4 scales. Through the double-branch structure, the feature representation containing context information and semantic information is obtained to calculate the image category probability, which further improves the accuracy of geographic positioning. On the University-1652, CVUSA and CVACT three datasets, the method of the technical solution has significantly improved the Recall@K and AP. In addition, the technical solution can also well exclude the interference of similar buildings and retrieve the correct image.

[0121] References

[0122] [1] Ahmed K T, Ummesafi S, Iqbal A. Content-based image feature information fusion retrieval. Information Fusion, 2019, 51: 76-99.

[0123] [2] Saritha R R, Paul V, Kumar P G. Content-based deep learning image retrieval. Cluster Computing, 2019, 22(2): 4187-4200.

[0124] [3] Outay F, Mengash H A, Adnan M. Applications of unmanned aerial vehicles (UAVs) in road safety, traffic, and highway infrastructure management: Recent advances and challenges. Transportation research part A: policy and practice, 2020, 141: 116-129.

[0125] [4] Zhu H, Ma M, Ma W, et al. Spatial channel progressive fusion ResNet for remote sensing classification[J]. Information Fusion, 2021, 70: 72-87.

[0126] [5] Wang P, Fan E, Wang P. Comparative analysis of image classification algorithms based on traditional machine learning and deep learning[J]. Pattern Recognition Letters, 2021, 141: 61-67.

[0127] [6] Zhang D, Ye M, Liu Y, et al. Multi-source unsupervised domain adaptation object detection[J]. Information Fusion, 2022, 78: 138-148.

[0128] [7] Tan M, Pang R, Le Q V. Efficientdet: scalable and efficient object detection[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 10781-10790.

[0129] [8] Yuan Y, Chen X, Wang J. Object context representation for semantic segmentation[C] / / European conference on computer vision. Springer, Cham, 2020: 173-190.

[0130] [9] Hao S, Zhou Y, Guo Y. A review of semantic segmentation based on deep learning[J]. Neurocomputing, 2020, 406: 302-321.

[0131]

[10] Jaouedi N, Boujnah N, Bouhlel M S. A new hybrid deep learning model for human action recognition[J]. Journal of King Saud University-Computer and Information Sciences, 2020, 32(4): 447-453.

[0132]

[11] Yang C, Xu Y, Shi J, et al. Time pyramid network for action recognition[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 591-600.

[0133]

[12] Shi Y, Yu X, Liu L, et al. Optimal feature transmission for cross-view image geo-localization[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(07): 11990-11997.

[0134]

[13] Zheng Z, Wei Y, Yang Y. University-1652: Multi-view multi-source benchmark for geo-localization based on UAV[C] / / Proceedings of the 28th ACM international conference on Multimedia. 2020: 1395-1403.

[0135]

[14] Wang T, Zheng Z, Yan C, et al. Every part matters: Local patterns facilitate cross-view geo-localization[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(2): 867-879.

[0136]

[15] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.

[0137]

[16] Zhang Z, Lan C, Zeng W, et al. Global relation attention for person re- identification [C] / / Proceedings of the ieee / cvf conference on computer vision and pattern recognition. 2020: 3186-3195.

[0138]

[17] Yu F, Koltun V, Funkhouser T. Dilated residual networks [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 472-480.

[0139]

[18] Zheng Z, Zheng L, Yang Y. A discriminatively learned convolutional neural network embedding for person re-identification [J]. ACM transactions on multimedia computing, communications, and applications (TOMM), 2017, 14(1): 1-20.

[0140]

[19] Li X, Yu L, Chang D, et al. Dual cross-entropy loss for small sample fine-grained vehicle classification [J]. IEEE Transactions on vehicular technology, 2019, 68(5): 4204-4212.

[0141]

[20] Workman S, Jacobs N. Position dependence of convolutional neural network features [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2015: 70-78.

[0142]

[21] Workman S, Souvenir R, Jacobs N. Wide-area image geolocating using aerial reference imagery [C] / / Proceedings of the IEEE International Conference on Computer Vision. 2015: 3961-3969.

[0143]

[22] Lin T Y, Cui Y, Belongie S, et al. Deep representation learning for ground-to- aerial geolocalization [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 5007-5015.

[0144]

[23] Vo N N, Hays J. Positioning and orienting street view images using overhead imagery [C] / / European conference on computer vision. Springer, Cham, 2016: 494-509.

[0145]

[24] Tian Y, Chen C, Shah M. Cross-view image matching for geolocalization in urban environments [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 3608-3616.

[0146]

[25] Altwaijry H, Trulls E, Hays J, et al. Learning to match aerial images and deep focus aware architecture [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 3539-3547.

[0147]

[26] Zhai M, Bessinger Z, Workman S, et al. Predicting ground scene layout from aerial images [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 867-875.

[0148]

[27] Hu S, Feng M, Nguyen R M H, et al. Cvm-net: cross-view matching network for image-based geo-localization [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 7258-7267.

[0149]

[28] Arandjelovic R, Gronat P, Torii A, et al. NetVLAD: CNN architecture for weakly supervised location recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 5297-5307.

[0150]

[29] Shi Y, Liu L, Yu X, et al. Spatial perception feature aggregation for image-based cross-view geo-localization [C] / / Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019: 10090-10100.

[0151]

[30] Shi Y, Yu X, Campbell D, et al. Where am I looking? Joint location and orientation estimation via cross-view matching [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 4064-4072.

[0152]

[31] Liu L, Li H. Cross-view geo-localization with orientation in neural networks[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 5624-5633.

[0153]

[32] Rodrigues R, Tani M. Are these the same place? Seeing unseen image geo-localization in cross-view[C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2021: 3753-3761.

[0154]

[33] Regmi K, Shah M. Bridging the domain gap for matching ground and aerial images[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 470-479.

[0155]

[34] Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144.

[0156]

[35] Toker A, Zhou Q, Maximov M, et al. Coming to earth: Geo-localization of satellite to street view synthesis[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 6488-6497.

[0157]

[36] Zheng Z, Zheng L, Garrett M, et al. Dual-path convolutional image text embedding with instance loss. ACM transactions on multimedia computing, communications, and applications (TOMM), 2020, 16(2): 1-23.

[0158]

[37] Ding L, Zhou J, Meng L, et al. A practical cross-view image matching method for unmanned aerial vehicles and satellites based on unmanned aerial vehicle geolocating. Remote Sensing, 2020, 13(1): 47.

[0159]

[38] Zhuang J, Dai M, Chen X, et al. A more efficient cross matching method for unmanned aerial vehicle and satellite images. Remote Sensing, 2021, 13(19): 3979.

[0160]

[39] Lin J, Zheng Z, Zhong Z, et al. Joint representation learning and keypoint detection for cross-view geolocating. IEEE Transactions on Image Processing, 2022.

[0161]

[40] Dai M, Hu J, Zhuang J, et al. Transformer-based unmanned aerial vehicle view geolocating feature segmentation and region alignment method. IEEE Transactions on Circuits and Systems for Video Technology, 2021.

[0162]

[41] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Advances in neural information processing systems, 2017, 30.

[0163]

[42] Chechik G, Sharma V, Shalit U, et al. Large-scale image similarity online learning based on ranking [J]. Journal of Machine Learning Research, 2010, 11(3).

[0164]

[43] Cai S, Guo Y, Khan S, et al. Geo-localization of aerial image pairs with triple loss and hard sample reweighting [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 8391-8400.

[0165]

[44] Hu J, Shen L, Sun G. Distillation and excitation network [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7132-7141.

Claims

1. A cross-view geolocalization method globally relation attention guided, characterized in that Comprising the following steps: A deep residual network is used as the backbone network, and a global relationship attention module is used to capture more robust global structure information of the image for matching; a double-branch network including a global branch and a local branch is designed to capture deep features with rich semantic information and local features with multi-scale context information respectively; in the local branch, dilated convolution is used to increase the receptive field of the feature map, and a square ring division strategy is used to divide the feature map at four scales; for the feature maps of each branch, the feature maps of each branch are converted into column vector descriptors, and then a classifier is used to obtain the predicted class of each column vector; a cross-entropy loss function is used to measure the difference between the predicted class and the real class of the image; The structure of the double-branch network is divided into a global branch and a local branch, which share network weights; first, ResNet50 is used as the backbone network, and the average pooling layer and the classification layer are removed to extract the features of the input image; at the same time, a global relationship attention module is added after extracting the shallow features, including global spatial relationship attention and global channel relationship attention, to capture the global structure information of the image; second, the double-branch structure is used to process the output features of the previous stage, effectively focusing on global and local information; The global branch is used to extract high-level semantic information of the whole image; the local branch focuses on deep features of the network, thereby retaining more image detail information; at the same time, in order to combine the information of the surrounding area of the target, the square ring division strategy is used in the local branch to divide the feature map into four different regions; finally, the global average pooling is used to convert the high-level features of the image into column vector descriptors; in the training stage, the classifier module is used to obtain the predicted class probability of each column vector descriptor, and the cross-entropy loss function is used to minimize the difference between the predicted class and the real class; the Euclidean distance is used to calculate the similarity between the query image and the database image in the test process, and finally the retrieval results are sorted based on the similarity.

2. The global relational attention-guided cross-view geolocalization method according to claim 1, characterized in that Comprising the following steps: The global relationship attention module is combined with the feature extraction network, including global spatial relationship attention and global channel relationship attention, a global relationship attention guided feature extraction network is constructed, and the attention weight is calculated by learning the relationship between the feature nodes; for the feature vector in the feature map, the feature vector is represented as a feature node x i , wherein i=1, 2, LN, and N is the number of feature nodes; for a certain feature node x i , the correlation r i,j between the current node and other nodes is calculated j,i , wherein j=1, 2, LN, and thus the relationship vector of the feature node x i is r i =[r i,1 , r i,2 , L, r i,N , r 1,i , r 2,i , L, r N,i ]; then, the feature node is spliced with the relationship vector r i , and a relationship-aware feature is obtained, so as to infer the attention weight of the current feature node.

3. The global relational attention-guided cross-view geolocalization method according to claim 2, characterized in that Comprising the following steps: The global spatial relationship attention learns the correlation between all feature nodes in the spatial dimension of the feature map, so that the network captures the features of the salient target region; Specifically, for the feature map obtained from the neural network The C-dimensional feature vector of each spatial position is taken as a feature node, forming a graph G with N = W x H nodes in common S The feature node is denoted as x i Where i = 1, 2, L N; the feature node x i The correlation r j between x i,j is obtained through a dot product operation, and is specifically defined as formula (1): r i,j = f s (x i ,x j ) = (ReLU(BN(Conv(x i )))) T (ReLU(BN(Conv(x j )))) (1) where f s (·) denotes the dot product operation, ReLU(·) denotes the rectified linear unit activation function, BN(·) denotes the batch normalization layer, Conv(·) represents the 1 x 1 convolution operation, and d is a pre-defined positive integer to control the dimension reduction ratio; similarly, the correlation between feature nodes x j and x i is obtained as r j,i , and the pairwise relationship between feature nodes x i,j and x j,i is represented as (r i , r j ); finally, the correlation between all nodes is represented by the relationship matrix R , where r i,j = R S (i,j). Stacking the correlation between the i-th feature node and all nodes in a fixed order to obtain a spatial relationship vector wherein R S (i,:) represents the correlation between the i-th feature node and all nodes, R S (:,i) represents the relationship between all nodes and the i-th node; in order to make full use of the global structure information of the feature node, the spatial relationship vector r i is spliced with the feature node x i itself to obtain a spatial relationship perception feature E S , defined as formula (2): E S = C(x i , r i ) = (pool c (ReLU(BN(Conv(x i )))), (ReLU(BN(Conv(r i ))))) (2) where C(·) denotes a concatenation operation, pool c (·) denotes a global average pooling in the channel dimension, reducing the channel dimension to 1; the spatial attention weight a i for the i-th feature is computed by the spatial relation-aware feature, defined as equation (3): a i = sigmoid(BN(Conv2(ReLU(BN(Conv1(E S )))))) (3) Where sigmoid(·) represents the sigmoid activation function, Conv2(·) converts the channel number to 1, and Conv1(·) reduces the dimension by a fixed ratio; The global channel relationship attention learns the relationship between all feature nodes in the channel dimension of the feature map, and gives different weights to each channel; Specifically, for the feature map The feature map on each channel is taken as a feature node to form a graph G with C nodes in common C Each feature node is represented as x i Where i = 1, 2, L C; For the input feature map S, it is compressed spatially into The feature node x i is obtained j The correlation r i,j between the feature node x j is defined as formula (4): r i,j = f c (x i , x j ) = (ReLU(BN(Conv(x i )))) T (ReLU(BN(Conv(x j )))) (4) Where f c (·) represents the dot product operation; similarly, we obtain the feature node x. j With x i The correlation between them r j,i Using matrices This represents the correlation between all nodes; the channel relationship vector is obtained by stacking the correlations between the i-th feature node and all nodes. And obtain the channel relationship feature E C and channel attention weight a i .

4. The global relational attention-guided cross-view geolocalization method according to claim 1, characterized in that Comprising the following steps: Dilated convolution structure is introduced in the deep residual network, which expands the receptive field by inserting r-1 weight values of 0, where r is the dilation factor, which is 1 in standard convolution operation; using a 3x3 convolution kernel under the same conditions, the receptive fields of standard convolution and dilated convolution are 3x3 and 5x5 respectively; For the residual block of the local branch network, the dilation convolution operation with dilation factor of 2 and 4 is used to increase the receptive field of the feature map; at the same time, the step of the convolution layer and the down-sampling layer in the last residual block of ResNet50 is adjusted to 1, when the resolution of the input image is 256*256, the resolution of the feature image output by the backbone network is 8*8, and the resolution of the feature image output by the dilated residual network is 32*32; In order to better distinguish the images in different geographical positions, the environment around the target building is taken as auxiliary information, and a square ring division strategy is adopted in the local branch to segment the feature image, which is divided into four parts according to the distance to the center of the image, and the feature images of different regions are obtained; then, the image features are converted into 2048-dimensional feature vectors through average pooling operation, which is represented by formula (5): where Avgpool(·) represents an average pooling operation, denotes the divided local branch feature map in different view platforms, represent the 4 local branch 2048-dimensional feature vectors after pooling.

5. The global relational attention-guided cross-view geolocalization method according to claim 1, characterized in that The method comprises the following steps: In the global branch, a deep residual network is used to extract and refine large-scale features, so as to obtain a feature map f containing rich semantic information j Thus, the categories to which different image features belong are identified; then, the global feature map is processed by average pooling to obtain a 2048-dimensional feature vector, which is represented by formula (6): g j = Avgpool(f j ) (6) where g j represents the global branch feature vector after pooling.

6. The global relational attention-guided cross-view geolocalization method according to claim 1, characterized in that The method comprises the following steps: The classifier is fused after the feature extraction stage to predict the class of each feature vector; the classifier is composed of a fully connected layer, a batch normalization layer, a Dropout layer and a classification layer; the local feature vector of the image and the global feature vector g j are input to predict the class to which each feature vector belongs, and finally the local prediction probability distribution vector of the image and the global prediction probability distribution vector q j are obtained. At the same time, the cross-entropy loss function is used to measure the distribution difference between the predicted probability and the real probability of the image; the cross-entropy loss is represented by formula (7): wherein denotes the corresponding original image after the processing of the square ring division strategy, x j denote the input images, j = 1 denotes the unmanned aerial vehicle platform, and j = 2 denotes the satellite platform; y denotes the real category of the input image, respectively denote and x j the normalized probability score belonging to the real category, defined by equation (8) and equation (9), Wherein C represents the number of all geographical label categories in the database.

Images