Cross-view geolocalization method based on non-local information attention mechanism

Abstract

The application discloses a cross-view geographic positioning method based on a non-local information attention mechanism. The non-local information attention mechanism is applied to a cross-view geographic positioning task. The non-local information attention mechanism model fuses global information, combines feature information of a context, and alleviates problems such as a limited field of view in training, loss of part of information, and inaccurate capture of key features. In addition, the non-local information attention mechanism to a certain extent improves the problem of a large difference between feature distributions of different views of the same geographic building in the cross-view geographic positioning task, optimizes a training model, thereby improving the accuracy and efficiency of matching retrieval and classification, and the non-local information attention mechanism can be easily embedded into many existing models, further improving the performance of the model.

Description

Technical Field

[0001] This invention belongs to the field of cross-perspective geolocation, specifically involving a cross-perspective geolocation method based on a nonlocal information attention mechanism, which matches and retrieves geographic target images from different perspectives, and extends to the problem of positioning and navigation. Background Technology

[0002] With the evolution of technology, the market demand for multi-view geolocation technology is increasing, and the application of cross-view matching to positioning and navigation is receiving more and more attention. This involves retrieving corresponding geographic targets from images from different viewpoints. Currently, this research is gradually being applied to fields such as autonomous driving, drone delivery, robotic transportation, and autonomous inspection. Cross-view geolocation refers to inferring geographic location from images from different viewpoints and is generally considered an image retrieval task. Typically, given a drone viewpoint image, the goal is to find images of the same target object from a candidate image library of satellite images. We can then utilize the relevant information in the satellite images, combined with existing positioning technologies, to achieve functions such as navigation. Furthermore, when GPS signals are relatively weak or unstable, it can be used as an independent auxiliary tool to help users locate themselves. With the maturity of deep learning technology, it has been widely applied in the field of computer vision, and neural networks have been introduced into cross-view geolocation. Therefore, the performance of the model has been significantly improved. For example, a pre-trained neural network model can be used to extract relevant features from cross-view geographic images. Through training, relevant semantic information can be extracted, and important features can be retained. Therefore, researchers can extract rich semantic information about geographic targets in images to achieve multi-view image matching and ultimately complete the task of retrieving location targets. During feature extraction, it is crucial to pay attention to both local and global information; combining this with contextual semantic information helps improve retrieval and classification performance.

[0003] Attention mechanisms based on nonlocal information are widely used in neural network machine learning and have achieved good results in many scenarios. Research applying visual attention mechanisms to deep learning often uses masks to form the attention mechanism. The purpose of a mask is to obtain a new layer of weights, calculate similarity, and extract key information from the image, which is often crucial for performance improvement. Through model training, relevant regions of interest are extracted from the image, i.e., the attention mechanism. In some scenarios, capturing long-range feature dependencies requires accumulating many network layers, which reduces learning efficiency. Furthermore, if the network is deep, careful design of modules and gradients is necessary. Nonlocal operations are computationally efficient, often requiring fewer stacked layers to achieve the same effect. Nonlocal operations can ensure that the input and output scales remain unchanged, and this design can be easily embedded into current network architectures. Different methods exist for calculating image similarity and extracting key information, resulting in varying performance of nonlocal attention mechanisms. In specific solutions, it is essential to utilize contextual information as much as possible and improve the model's robustness.

[0004] In today's fast-paced world, people's demands for efficient delivery services like express delivery and food delivery are increasing, requiring these industries to maximize efficiency. This necessitates higher levels of automation, which inevitably places higher demands on geolocation navigation technology. Drone technology is currently experiencing rapid development, and its integration with geolocation technology offers significant research potential and a vast application market. For example, in the context of the pandemic, drones and robots can be used for contactless delivery, greatly reducing the risk of infection. In disaster relief, unmanned delivery will significantly improve efficiency, enabling precise point-to-point material delivery and mitigating risks for rescue personnel. However, in cross-perspective geolocation tasks, the differences between satellite and drone images lead to significant variations in feature distributions learned by neural networks, sometimes hindering successful retrieval and classification. Currently, many research methods exist in this field. Some employ deep networks, which significantly reduces learning efficiency and hinders product deployment. Other methods focus too much on local information, neglecting useful contextual information, limiting their scope and thus requiring further improvement in accuracy and model performance. Similarly, during the matching process of images from different perspectives, there are some information misalignments, which is not conducive to the retrieval of buildings. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a cross-perspective geolocation method based on a nonlocal information attention mechanism.

[0006] To better extract global information, fully utilize both global and local information, and improve the dependencies between arbitrary elements in the global context, this invention considers improving the neural network using an attention mechanism based on non-local information. This extracts key information, reduces the loss of useful information, and integrates contextual information to alleviate information misalignment in image patches, increase the focus on important features, and further enhance the performance of geographic target localization.

[0007] Cross-perspective geolocation suffers from problems such as insufficient mining of image feature information, loss of some global information, insufficient interaction of contextual information, and excessive differences in the distribution of image features from different perspectives. These problems can be alleviated by using a non-local information attention mechanism module.

[0008] This invention applies a nonlocal information attention mechanism to the field of cross-view geolocation. By utilizing this mechanism, global information can be better integrated, promoting feature alignment between images from different perspectives, increasing the attention to key features, and optimizing feature distribution. This makes it easier to align image features from different perspectives, improving the similarity of images of the same geographic location, and thus more effectively improving retrieval efficiency and accuracy. By using this nonlocal information attention mechanism, the feature distribution of images from different perspectives is altered, incorporating surrounding information. Important features are enhanced, resulting in better classification performance and improved matching accuracy for the same target across cross-view images. In implementing the nonlocal information attention mechanism, this invention employs self-attention and mutual attention operations on the feature parts, weighted summing to obtain a new feature matrix, optimizing the feature map, more fully mining key information, and improving model performance.

[0009] The cross-perspective geolocation method based on nonlocal information attention mechanism includes the following steps:

[0010] Step 1: Use the ResNet-50 network model to extract features from the drone's viewpoint and the satellite's viewpoint, respectively.

[0011] Step 2: Use the square ring segmentation method to divide the features. Based on the distance from the image to the image center, divide the image into eight parts. After separation, perform average pooling and fuse contextual information. The center of the square ring partition can be approximated as the center of the feature map. The eight feature parts eventually become matrices with eight identical dimensions, among which the channel dimension information is more important.

[0012] Step 3: Using the nonlocal information attention mechanism module, generate the self-attention weights for each feature part, and multiply them with the original feature matrix to obtain the feature matrix after self-attention operation, i.e., the self-attention features.

[0013] Step 4: Each feature part from Step 2 then performs mutual attention operations with its two nearest neighbor feature parts, generating two cross-correlation matrix weights. These weights are then multiplied by their corresponding feature matrices to obtain the mutual attention features. A multi-head mechanism is introduced during the calculation of self-attention and mutual attention to improve computational efficiency.

[0014] Step 5: The self-attention features obtained in Step 3 and the mutual attention features obtained in Step 4 are weighted and summed to obtain the final new feature matrix, i.e., the new feature map. This optimizes the model, integrates global information, and improves network performance.

[0015] The beneficial effects of this invention are as follows:

[0016] This invention applies a non-local information attention mechanism to cross-perspective geolocation tasks. By fusing global information with contextual features through a non-local information attention mechanism model, it alleviates problems such as insufficient field of view, partial information loss, and inaccurate key feature capture during training of local features. Furthermore, it improves the issue of significant differences in feature distribution across different perspectives of the same geographic building in cross-perspective geolocation tasks, optimizing the training model and thus improving the accuracy and efficiency of matching retrieval and classification. Moreover, the non-local information attention mechanism can be easily embedded into many existing models, further enhancing model performance. Attached Figure Description

[0017] Figure 1 This is a structural diagram of the nonlocal information attention mechanism module in an embodiment of the present invention;

[0018] Figure 2 This invention relates to a cross-perspective geolocation network based on a nonlocal information attention mechanism, as described in this embodiment. Detailed Implementation

[0019] This invention proposes a cross-perspective geolocation method based on a nonlocal information attention mechanism. The nonlocal information attention mechanism module structure is as follows: Figure 1 As shown, the cross-perspective geolocation network process based on the nonlocal information attention mechanism is as follows: Figure 2 As shown.

[0020] The specific steps are as follows:

[0021] For example, the images in the drone and satellite view datasets used in training are both set to 256*256*3 pixels, with a given batch size, and the dataset is screened before training to remove non-standard images.

[0022] Step 1: Using the ResNet-50 neural network model, extract features from satellite and drone view images respectively. In the specific design, the final pooling and classification layers in the original ResNet-50 network model are removed, resulting in feature maps for satellite and drone view images with dimensions of batch*16*16*2048.

[0023] Step 2: Perform circular segmentation on the feature maps obtained in the previous step, dividing the global features of both perspectives into eight parts, resulting in features of size batch*h*w*2048. Next, perform average pooling on the eight feature matrices of the satellite and UAV perspective images respectively, resulting in eight new feature matrices for each perspective image. After adjusting the dimensions, pass them through a linear layer with 512 output channels. Finally, the dimension of each feature matrix is ​​adjusted to batch*512*1*1. Here, the eight new feature matrices for the same perspective can be represented as fi (i∈{1,2,3,4,5,6,7,8}), where the superscript i represents the i-th part from the center. To distinguish between the two perspectives, the newly generated features of the satellite and UAV perspectives are represented as follows: and

[0024] Step 3: The eight feature matrices obtained in Step 2 from the satellite and UAV view images are fed into the nonlocal information attention mechanism module. In the nonlocal information attention module, the eight feature parts first undergo self-attention operations to form corresponding mask layers, i.e., feature map weights. Using feature matrix f... 2 For example, first perform self-attention operation to generate the mask layer. 22 , represented as M 22 Set the weight M 22 With the original feature f 2 Multiplying them together yields the new feature matrix t after the self-attention operation. 22 .

[0025] Step 4: Each feature part then performs mutual attention operations with its two nearest neighbor feature parts. Similarly, using the feature matrix f... 2 For example, f 2 Then respectively with f 1 and f 3 Perform mutual attention operations to generate a mask layer. 21 and Mask 23 , respectively denoted as M 21 M 23 And the weight M 21 With the original feature f 1 Multiplication, weight M 23 With the original feature f 3Multiplying them together yields a new feature matrix t after the mutual attention operation. 21 , t 23 In the computation of self-attention and mutual attention, a multi-head mechanism is introduced to improve computational efficiency.

[0026] Step 5: Assign a linear weighting coefficient to the obtained self-attention new feature matrix and mutual attention new feature matrix respectively, and then sum them up to obtain the final feature matrix. For example, the feature matrix t generated in steps 3 and 4... 21 , t 22 , t 23 The weighted summation yields the final feature matrix F2. To make the model more stable, a simple MLP layer is added after F2. After the above process, eight new features Fi (i∈{1,2,3,4,5,6,7,8}) are obtained for each of the two viewpoints. Similarly, to distinguish between the two viewpoints, the new features for the satellite viewpoint and the drone viewpoint are represented as follows: and During the computation process, a multi-head mechanism is introduced to improve computational efficiency, and the generated self-attention and mutual attention features are normalized using Softmax to enhance the model's robustness. Finally, the new features are fed into a linear classifier, the complete model is loaded, and the geographic target retrieval task is completed.

[0027] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0028] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A cross-view geolocating method based on non-local information attention mechanism, characterized in that, Comprising the following steps: Step one: using ResNet-50 network model, respectively extracts the features of unmanned aerial vehicle perspective and satellite perspective image; Step two: adopt square ring segmentation method to divide the features, according to the distance from the image to the center of the image, the image is divided into eight parts, after separation average pooling, fusion context information, the center of square ring partition is approximately regarded as the center of feature mapping, eight feature parts finally become eight matrices with the same dimension, among which the information of channel dimension is more important; Step three: use non-local information attention mechanism module to generate self-attention weight of each feature part, and multiply it with the original feature matrix to get the feature matrix after self-attention operation, that is, self-attention feature; Step four: each feature part in step two is respectively calculated with the nearest two feature parts to generate two mutual correlation matrix weights, and is multiplied with the corresponding feature matrix to get mutual attention feature; In the calculation process of self-attention and mutual attention, multi-head mechanism is introduced to improve the calculation efficiency; Step five: the self-attention feature obtained in step three and the mutual attention feature obtained in step four are weighted and summed to obtain the final new feature matrix, that is, the new feature map; Optimize the model, fuse global information and improve the network performance.

2. The cross-view geolocating method based on non-local information attention mechanism according to claim 1, characterized in that, The specific method of step one is as follows: Using ResNet-50 neural network model, respectively extracts the features of satellite perspective image and unmanned aerial vehicle perspective image; Remove the last pooling layer and classification layer of the original ResNet-50 network model, finally respectively get the feature map of satellite perspective image and unmanned aerial vehicle perspective image, the dimension is batch*16*16*2048.

3. The cross-view geolocating method based on non-local information attention mechanism according to claim 2, characterized in that, The specific method of step two is as follows: The feature map obtained in step one is circularly segmented, and the global features of the two perspectives are respectively segmented into eight parts to obtain features of corresponding batch*h*w*2048 size; then average pooling is performed on the eight feature matrices of the satellite perspective image and the unmanned aerial vehicle perspective image respectively, thereby obtaining eight new feature matrices of each perspective image, adjusting the dimensions to pass through a linear layer with an output of 512 channels, and finally adjusting the dimension of each feature matrix to batch*512*1*1, where the eight new feature matrices of the same perspective can be represented as f i , i∈{1,2,3,4,5,6,7,8}, where the superscript i represents the ith part from the center; in order to distinguish between the two perspectives, the newly generated features of the satellite perspective and the unmanned aerial vehicle perspective are respectively represented as and 4. The cross-view geolocating method based on non-local information attention mechanism according to claim 3, characterized in that, The specific method of step three is as follows: The eight feature matrices obtained in step 2 from the satellite perspective and the unmanned aerial vehicle perspective images are respectively sent into a non-local information attention mechanism module; in the non-local information attention module, the eight feature parts are first respectively subjected to self-attention operation to form corresponding mask layers Mask, that is, feature map weights; the feature matrix f 2 is taken as an example, first subjected to self-attention operation to generate a mask layer Mask 22 , denoted as M 22 ; the weight M 22 is multiplied by the original feature f 2 to obtain a new feature matrix t 22 after self-attention operation.

5. The cross-view geolocating method based on non-local information attention mechanism according to claim 4, characterized in that, The specific method of step four is as follows: Each feature part is simultaneously subjected to mutual attention operation with the two most adjacent feature parts; similarly, the feature matrix f 2 For example, f 2 is subjected to mutual attention operation with f 1 and f 3 , respectively, to generate mask layers Mask 21 and Mask 23 , denoted as M 21 and M 23 , respectively; and the weight M 21 is multiplied with the original feature f 1 , and the weight M 23 is multiplied with the original feature f 3 , to obtain new feature matrices t 21 and t 23 after mutual attention operation; in the calculation process of self-attention and mutual attention, a multi-head mechanism is introduced to improve the calculation efficiency.

6. The cross-view geolocating method based on non-local information attention mechanism according to claim 5, characterized in that, The specific method of step five is as follows: The self-attention new feature matrix and the mutual-attention new feature matrix are respectively given a linear weighting coefficient, and a final feature matrix is obtained after weighted summation; the feature matrices t 21 , t 22 , t 23 are weighted and summed to obtain a final feature matrix F2; in order to make the model more stable, a simple MLP layer is connected behind F2; after the above steps, eight new features F i , i∈{1,2,3,4,5,6,7,8} of two view images are finally obtained; similarly, in order to distinguish the two views, the new features of the satellite view and the unmanned aerial vehicle view are respectively represented as and In the calculation process, a multi-head mechanism is introduced to improve the calculation efficiency, and the generated self-attention and mutual-attention features are normalized by Softmax to enhance the robustness of the model; finally, the new features are connected to a linear classifier, the complete model is loaded, and the geographic target retrieval task is completed.

Images