Cross-view geolocation method based on unit dot product attention mechanism
By embedding a unit dot product attention module into a convolutional neural network, the salient region information of the image is enhanced, which solves the problem that the salient location information of the image is not mined in existing methods, and achieves higher cross-view geolocation accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HARBIN INST OF TECH
- Filing Date
- 2024-05-07
- Publication Date
- 2026-05-26
AI Technical Summary
Existing cross-view geolocation methods lack the ability to extract salient location information from images, which leads to a bottleneck in improving the performance of cross-view geolocation tasks.
We employ a unit dot product attention mechanism to enhance the salient region information of the feature map by embedding unit dot product attention modules at different depths of the convolutional neural network. This is combined with a geographic location classification agent learning task to optimize viewpoint-invariant image-level feature extraction.
It improves the accuracy and robustness of cross-view geolocation and significantly enhances the ability to extract view-invariant image-level features.
Smart Images
Figure CN118261970B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a cross-view geolocation method, specifically a cross-view geolocation method based on a unit dot product attention mechanism. Background Technology
[0002] Cross-view geolocation aims to estimate the location of an image by matching a query image with a geotagged image database collected from different viewpoints. Cross-view geolocation has been applied to many real-world tasks, such as drone navigation and drone delivery. Due to the diversity of image viewpoints and capture times, extracting highly discriminative view-invariant image-level features for cross-view geolocation is quite challenging. Most existing methods typically extract view-invariant image-level features focusing on the overall image or large-scale structure, lacking the ability to mine salient location information within the image. This limits their performance improvement in cross-view geolocation tasks. Summary of the Invention
[0003] To fully exploit the fine-grained information in images and extract highly discriminative view-invariant image-level features, this invention provides a cross-view geolocation method based on a unit dot product attention mechanism.
[0004] The objective of this invention is achieved through the following technical solution:
[0005] A cross-view geolocation method based on a unit dot product attention mechanism includes the following steps:
[0006] Step 1: Extract shallow feature maps from different perspectives:
[0007] Images from three channels (high H, width W) representing satellite, ground, and drone perspectives. The inputs are fed into ResNet networks B, which are pre-trained on the ImageNet dataset for each viewpoint. S B G B D The first stage is to obtain feature maps.
[0008] Step 2: Enhance shallow feature maps from different perspectives using the unit dot product attention module:
[0009] For each viewpoint v∈{S,G,D}, the image I v The feature map obtained in step one The input is fed into the unit dot product attention module to compute the attention-enhanced feature map. The specific steps are as follows:
[0010] Step 2.1: Given a feature map First, input it into a convolution kernel of size 1×1 and number C. d In a two-dimensional convolutional layer, L2 regularization is then applied to obtain a depth descriptor map. The calculation process is expressed as follows:
[0011]
[0012] Where Conv1 represents a convolution operation with a kernel size of 1×1;
[0013] Step 22: In the depth descriptor graph Perform unit dot product convolution operations on the top:
[0014] right For each spatial position (i,j), use the element at that position. Stacking produces a 3×3 convolution kernel, and a convolution operation is performed centered on this element, then inverted. The output depth saliency plot is denoted as . The calculation process is expressed as follows:
[0015]
[0016] Where A = {-1, 0, 1};
[0017] Steps two and three: The obtained... The keypoint attention mask is obtained by performing a non-linear mapping using the Sigmoid activation function. The above two calculation processes are expressed as follows:
[0018]
[0019] Where UDPC represents the unit dot product convolution operation in step 22, and K represents the size of the convolution kernel. In this invention, K = 3.
[0020] Step 24: Calculate the keypoint attention mask With feature map Residual attention fusion feature map
[0021] First of all The number of channels has been expanded to the same level as Consistent, and then through with Perform element-wise multiplication pairs We perform weighting to obtain attention-weighted feature maps. This process can be represented as:
[0022]
[0023] Where Expand is the channel expansion function;
[0024] After obtaining the attention-weighted feature map Then, it is compared with the feature map. The residual attention fusion feature map is obtained by adding them together. This process can be represented as:
[0025]
[0026] Step 3: Extract shallow feature maps from different perspectives:
[0027] Fuse feature maps with residual attention from each viewpoint The input is fed into the ResNet network B pre-trained on the ImageNet dataset corresponding to each viewpoint. v In the second stage, feature maps are obtained.
[0028] Step 4: Enhance deep feature maps from different perspectives using the unit dot product attention module:
[0029] The feature maps of each viewpoint v obtained in step three The input is fed into the unit dot product attention module to compute the attention-enhanced feature map. The specific steps are as follows:
[0030] Step 41: Given a feature map First, input it into a convolution kernel of size 1×1 and number C. d In a two-dimensional convolutional layer, L2 regularization is then applied to obtain a depth descriptor map. The calculation process is expressed as follows:
[0031]
[0032] Where Conv1 represents a convolution operation with a kernel size of 1×1;
[0033] Step 4.2, in the depth descriptor graph Perform unit dot product convolution operations on the top:
[0034] right For each spatial position (i,j), use the element at that position. Stacking produces a 3×3 convolution kernel, and a convolution operation is performed centered on this element, then inverted. The output depth saliency plot is denoted as . The calculation process is expressed as follows:
[0035]
[0036] Where A = {-1, 0, 1};
[0037] Step 43, the obtained The keypoint attention mask is obtained by performing a non-linear mapping using the Sigmoid activation function. The above two calculation processes are expressed as follows:
[0038]
[0039] Where UDPC represents the unit dot product convolution operation in step four-two, and K represents the size of the convolution kernel. In this invention, K = 3.
[0040] Step 4: Calculate the keypoint attention mask With feature map Residual attention fusion feature map
[0041] First of all The number of channels has been expanded to the same level as Consistent, and then through with Perform element-wise multiplication pairs We perform weighting to obtain attention-weighted feature maps. This process can be represented as:
[0042]
[0043] Where Expand is the channel expansion function;
[0044] After obtaining the attention-weighted feature map Then, it is compared with the feature map. The residual attention fusion feature map is obtained by adding them together. This process can be represented as:
[0045]
[0046] Step 5: Extract view-invariant image-level features from images at different viewpoints:
[0047] The residual attention fusion feature map of each view v∈{S,G,D} is used to fuse the feature map. The input is fed into the ResNet network B pre-trained on the ImageNet dataset corresponding to each viewpoint. v In the third, fourth, and fifth stages, feature maps are obtained. and feature maps Perform global average pooling to obtain the image-level features of each viewpoint v.
[0048] Step Six: Optimize the model's ability to extract viewpoint-invariant image-level features through a geolocation classification agent learning task. The specific steps are as follows:
[0049] Step 61: Extract the image-level features (ILF) from the viewpoint v∈{S,G,D} v The input is fed into a classifier ClsHead, which consists of a 512-dimensional fully connected layer (FC), a batch normalization layer (BN), and a fully connected classification layer (Cls) connected in series, and the image I is obtained through the Softmax function. v The probability of belonging to each of the N geographic location categories. The above process can be represented as follows:
[0050] p(I v =Softmax(ClsHead(ILF) v ))
[0051] Step 62, using Instance loss L instance Optimize this category task:
[0052] L instance =-log(p(I v )[y])
[0053] Where y is image I v The geographical location category to which it belongs.
[0054] Compared with the prior art, the present invention has the following advantages:
[0055] This invention designs a cross-view geolocation method based on a unit dot product attention mechanism. The unit dot product attention mechanism, embedded at different depths in a convolutional neural network (CNN), can mine salient region information of different granularities in images, providing the CNN with feature maps containing rich fine-grained semantic information and improving the overall view-invariant image-level feature extraction capability of the neural network. Compared with existing cross-view geolocation methods based on ordinary CNNs, the accuracy of cross-view geolocation is significantly improved, demonstrating good localization robustness. Attached Figure Description
[0056] Figure 1 The flowchart shows a cross-view geolocation method based on the unit dot product attention mechanism.
[0057] Figure 2 A schematic diagram of unit dot product convolution operation;
[0058] Figure 3 Example of a keypoint attention mask. Detailed Implementation
[0059] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.
[0060] This invention provides a cross-view geolocation method based on a unit dot product attention mechanism. Inspired by the human visual system (HVS)'s process of distinguishing similar targets and keypoint detection and matching, this method embeds unit dot product attention modules at different depth positions in a standard convolutional neural network (CNN), improving the overall view-invariant image-level feature extraction capability of the neural network. Specifically, this invention designs a unit dot product convolution operation, a unit dot product attention module, and a cross-view geolocation framework suitable for CNNs. The unit dot product attention module discovers salient regions of the depth feature map at different depth positions in the CNN through the unit dot product convolution operation, providing the CNN with feature maps containing rich, fine-grained semantic information. Furthermore, this method uses geographic location classification as a proxy learning task to optimize the model's ability to extract view-invariant image-level features. Given a cross-view image database with geographic location labels, this invention can accurately determine the geographic location of a query image and has achieved competitive results on multiple cross-view geolocation datasets.
[0061] Figure 1 The diagram shows the overall structure of this method, which can be broadly divided into four parts:
[0062] The first part is shallow feature extraction and enhancement. For example... Figure 1 As shown, firstly, three channels of images from satellite, ground, and UAV perspectives with a height of H and a width of W are displayed. The inputs are fed into ResNet networks B, which are pre-trained on the ImageNet dataset for each viewpoint. S B G B D The first stage yields shallow feature maps. Then, through the unit dot product attention module... Enhance it. For example... Figure 1 and Figure 2 As shown, for each viewpoint v∈{S,G,D}, the image I v The obtained feature map First, input the kernel size of 1×1 and the number of kernels of C. d In a two-dimensional convolutional layer, L2 regularization is then applied to obtain a depth descriptor map. Then in the depth descriptor graph Perform unit dot product convolution operation on: For each spatial position (i,j), use the element at that position. Stacking produces a 3×3 convolution kernel, and a convolution operation is performed centered on this element, then inverted. The output depth saliency plot is denoted as . Subsequently, the obtained The keypoint attention mask is obtained by performing a non-linear mapping using the Sigmoid activation function. Finally, calculate the keypoint attention mask. With feature map Residual attention fusion feature map First of all The number of channels has been expanded to the same level as Consistent, and then through with Perform element-wise multiplication pairs We perform weighting to obtain attention-weighted feature maps. After obtaining the attention-weighted feature map Then, it is compared with the feature map. The residual attention fusion feature map is obtained by adding them together.
[0063] The second part is deep feature extraction and enhancement. For example... Figure 1 As shown, for each viewpoint v∈{S,G,D}, the residual attention fusion feature map of each viewpoint is used. The input is fed into the ResNet network B pre-trained on the ImageNet dataset corresponding to each viewpoint. v In the second stage, deep feature maps are obtained. Then, through the unit dot product attention module... Enhance it. For example... Figure 1 and Figure 3 As shown, for each viewpoint v∈{S,G,D}, the image I v The obtained feature map First, input the kernel size of 1×1 and the number of kernels of C. d In a two-dimensional convolutional layer, L2 regularization is then applied to obtain a depth descriptor map. Then in the depth descriptor graph Perform unit dot product convolution operation on: For each spatial position (i,j), use the element at that position. Stacking produces a 3×3 convolution kernel, and a convolution operation is performed centered on this element, then inverted. The output depth saliency plot is denoted as . Subsequently, the obtained The keypoint attention mask is obtained by performing a non-linear mapping using the Sigmoid activation function. Finally, calculate the keypoint attention mask. With feature map Residual attention fusion feature map First of all The number of channels has been expanded to the same level as Consistent, and then through with Perform element-wise multiplication pairs We perform weighting to obtain attention-weighted feature maps. After obtaining the attention-weighted feature map Then, it is compared with the feature map. The residual attention fusion feature map is obtained by adding them together.
[0064] The third part is view-invariant image-level feature extraction. For example... Figure 1 As shown, the residual attention fusion feature map of each view v∈{S,G,D} is used. The input is fed into the ResNet network B pre-trained on the ImageNet dataset corresponding to each viewpoint. v In the third, fourth, and fifth stages, feature maps are obtained. and feature maps Perform global average pooling to obtain the image-level features of each viewpoint v.
[0065] The fourth part describes the ability of the model to extract viewpoint-invariant image-level features through a geographic location classification agent learning task. For example... Figure 1 The diagram first shows the image-level features ILF of the viewpoint v∈{S,G,D}. v The input is fed into a classifier ClsHead, which consists of a 512-dimensional fully connected layer (FC), a batch normalization layer (BN), and a fully connected classification layer (Cls) connected in series, and the image I is obtained through the Softmax function. v The probability of belonging to each of the N geographic location categories. Then use Instance loss L instance To optimize this category task:
[0066] L instance =-log(p(I v )[y])
[0067] Where y is image I v The geographical location category to which it belongs.
[0068] Experimental results
[0069] This invention trains the entire model using the University-1652 and CVUSA datasets respectively. University-1652 contains images from satellite, drone, and ground perspectives, while CVUSA contains images from both ground and satellite perspectives. To ensure fairness in the comparison, this invention follows the standard data partitioning of these two datasets for performance testing.
[0070] This invention is implemented in PyTorch and trained on hardware equipped with an Intel(R) 11700k CPU and an Nvidia A800 GPU. The model input is uniformly adjusted to 256×256, the kernel size for all unit dot-matrix convolution operations is set to 3×3, and the number of channels C in the depth descriptor map is... d The value is set to 128. Due to the similarity in viewpoint between satellites and drones, the ResNet feature extraction network B for both viewpoints... S B D Shared weights were used. Training was performed using the SGD optimizer with a momentum of 0.9 and weight decay of 5e-4. The batch size was 16. Training on the University dataset involved 360 epochs with an initial learning rate of 0.01, which changed to 0.001 starting at the 200th epoch. Training on the CVUSA dataset involved 120 epochs with an initial learning rate of 0.05, which changed to 0.005 starting at the 80th epoch.
[0071] During the performance testing phase, this invention uses the image-level features ILF of each viewpoint v∈{S,G,D} output by the model. v As a feature descriptor for images, cosine distance is used to measure the similarity between the query image and images in a geotagged image database. On the University dataset, this invention uses Recall@1 and mAP as evaluation metrics. On the CVUSA dataset, this invention uses Recall@1, Recall@5, Recall@10, and Recall@1% as rating metrics.
[0072] Tables 1 and 2 show the performance comparison of this invention and other cross-view geolocation methods on the University-1652 and CVUSA datasets, respectively. When adapting the framework of this invention to other methods, the model was trained according to the training configuration of the compared methods.
[0073] Table 1 shows the performance comparison on the University-1652 dataset.
[0074]
[0075] Table 2 shows the performance comparison on the CVUSA dataset.
[0076]
[0077]
[0078] As shown in Tables 1 and 2, this invention significantly and effectively improves the performance of various cross-view geolocation methods based on convolutional neural networks on the University-1652 and CVUSA datasets. LPN and GeoDTR are currently the best-performing methods on the University-1652 and CVUSA datasets, respectively, and their performance is further improved after applying this invention to both methods.
[0079] also, Figure 3 This presentation showcases visualizations of keypoint attention masks and their local maxima at different depths, generated by unit dot product attention modules. The local maxima are obtained by performing non-maximum suppression (NMS) on the keypoint attention mask. It can be seen that shallower unit dot product attention modules focus more on the edges of large-scale scenes, while deeper modules focus more on fine textures within the scene.
Claims
1. A cross-view geolocation method based on a unit dot product attention mechanism, characterized in that... The method includes the following steps: Step 1: Extract shallow feature maps from different perspectives: The height is Width is 3-channel satellite, ground, and UAV view images The inputs are fed into the ResNet networks pre-trained on the ImageNet dataset for each viewpoint. The first stage is to obtain feature maps. ; Step 2: Enhance shallow feature maps from different perspectives using the unit dot product attention module: For each perspective Image The feature map obtained in step one The input is fed into the unit dot product attention module to compute the attention-enhanced feature map. The specific steps are as follows: Step 2.1: Given a feature map First, input it into the convolution kernel with a size of Quantity is In a two-dimensional convolutional layer, L2 regularization is then applied to obtain a depth descriptor map. The calculation process is expressed as follows: in, Indicates the kernel size as Convolution operations; Step 22: In the depth descriptor graph Perform unit dot product convolution operations on the top: right Each spatial location Use the element here Stacking produces a A convolution kernel of size [size] is used, and a convolution operation is performed centered on this element, then the kernel is inverted. The output depth saliency plot is denoted as [example]. The calculation process is expressed as follows: in ; Steps two and three: The obtained... The keypoint attention mask is obtained by performing a non-linear mapping using the Sigmoid activation function. The above two calculation processes are expressed as follows: in This refers to the unit dot product convolution operation in step two. To represent the size of the convolution kernel; Step 24: Calculate the keypoint attention mask With feature map Residual attention fusion feature map : First of all The number of channels has been expanded to the same level as Consistent, and then through with Perform element-wise multiplication pairs We perform weighting to obtain attention-weighted feature maps. The process is represented as: in This is a channel expansion function; After obtaining the attention-weighted feature map Then, it is compared with the feature map. The residual attention fusion feature map is obtained by adding them together. The process is represented as: ； Step 3: Extract shallow feature maps from different perspectives: Fuse feature maps with residual attention from each viewpoint The input is fed into the ResNet network pre-trained on the ImageNet dataset corresponding to each viewpoint. In the second stage, feature maps are obtained. ; Step 4: Enhance deep feature maps from different perspectives using the unit dot product attention module: The various perspectives obtained in step three Feature map The input is fed into the unit dot product attention module to compute the attention-enhanced feature map. ; Step 5: Extract view-invariant image-level features from images at different viewpoints: Each perspective Residual attention fusion feature map The input is fed into the ResNet network pre-trained on the ImageNet dataset corresponding to each viewpoint. In the third, fourth, and fifth stages, feature maps are obtained. and the feature map Perform global average pooling to obtain the current viewpoints. Image-level features ; Step 6: Optimize the model's ability to extract viewpoint-invariant image-level features through a geolocation classification agent learning task.
2. The cross-view geolocation method based on unit dot product attention mechanism according to claim 1, characterized in that... The specific steps of step four are as follows: Step 41: Given a feature map First, input it into the convolution kernel with a size of Quantity is In a two-dimensional convolutional layer, L2 regularization is then applied to obtain a depth descriptor map. The calculation process is expressed as follows: in, Indicates the kernel size as Convolution operations; Step 4.2, in the depth descriptor graph Perform unit dot product convolution operations on the top: right Each spatial location Use the element here Stacking produces a A convolution kernel of size [size] is used, and a convolution operation is performed centered on this element, then the kernel is inverted. The output depth saliency plot is denoted as [example]. The calculation process is expressed as follows: in ; Step 43, the obtained The keypoint attention mask is obtained by performing a non-linear mapping using the Sigmoid activation function. The above two calculation processes are expressed as follows: in This refers to the unit dot product convolution operation in step four-two. To represent the size of the convolution kernel; Step 4: Calculate the keypoint attention mask With feature map Residual attention fusion feature map : First of all The number of channels has been expanded to the same level as Consistent, and then through with Perform element-wise multiplication pairs We perform weighting to obtain attention-weighted feature maps. The process is represented as: in This is a channel expansion function; After obtaining the attention-weighted feature map Then, it is compared with the feature map. The residual attention fusion feature map is obtained by adding them together. The process is represented as: 。 3. The cross-view geolocation method based on unit dot product attention mechanism according to claim 1, characterized in that... The specific steps of step six are as follows: Step 61: Change perspective Image-level features A classifier that shares weights between input views and consists of a 512-dimensional fully connected layer, a batch regularization layer, and a fully connected classification layer concatenated. In the middle, the image is obtained through the Softmax function. belong The probability of each category in the geographical location categories The above process can be represented as: Step 62: Using Instance Loss Optimize this category task: in, For image The geographical location category to which it belongs.