A multi-level feedback joint representation learning based cross-view geolocation method

Abstract

The application belongs to the technical field of computer vision, and discloses a cross-view geolocation method based on multi-level feedback joint representation learning. A cross-view geolocation network is established, which is a double-branch architecture, the basic architectures of the two branches are similar and the swin-b network weights are shared; each branch is composed of three forward stages, which are feature extraction, feature aggregation and feature classification respectively; in the feature extraction and feature aggregation stages, the swin-b network, the multi-level feedback network and the adaptive region elimination strategy are realized; the unmanned aerial vehicle view image and the satellite view image are respectively input into each branch of the cross-view geolocation network, and after the feature extraction and the feature aggregation, the feature classification is carried out through the classifier. The test is carried out on the University-1652 dataset, and the experimental results can prove that the network model proposed by the application is better than the most advanced algorithm at present, and the effectiveness of the algorithm is verified.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a cross-view geolocation method based on multi-level feedback joint representation learning. Background Technology

[0002] Cross-view geolocation aims to match the most similar images of the same geographic target from different viewpoints or platforms. Essentially, it can be understood as an image retrieval task from two different acquisition sources. Early cross-view geolocation tasks primarily studied the matching problem between terrestrial panoramic images and satellite imagery. However, the viewpoints and imaging methods of the images acquired from these two platforms differ significantly: terrestrial views are approximately perpendicular to the horizon, while satellite views are approximately parallel to the ground. Therefore, this presents a significant challenge to cross-view geolocation tasks. Furthermore, the development of unmanned aerial vehicle (UAV) technology has further expanded the application scenarios of cross-view geolocation. Currently, cross-view geolocation has been widely applied in many fields, such as agriculture, UAV navigation, event detection, and precision delivery. Therefore, research on geolocation from both UAV and satellite image perspectives is of great significance.

[0003] The paper *IEEE Transactions on Circuits and Systems for Video Technology, 1–14, 2023* captures rich discriminative information through cross-dimensional interaction, obtaining multiple feature representations to achieve comprehensive feature representation and improve model performance. However, focusing only on global information in the image while ignoring other surrounding information can significantly impact model performance, leading to insufficient model performance. The paper *IEEE Transactions on Circuits and Systems for Video Technology, 867–879, 2021* proposes that each feature in an image possesses a certain degree of discriminative power and uses a square ring segmentation method to achieve cross-view geolocation tasks. However, focusing only on local information in the image, while these ineffective surrounding information may not be crucial, can still affect the final model result, leading to unsatisfactory model performance. Summary of the Invention

[0004] The purpose of this invention is to propose a cross-view geolocation method based on multi-level feedback joint representation learning, which improves the accuracy and robustness of cross-view geolocation.

[0005] The technical solution of the present invention is as follows: a cross-view geolocation method based on multi-level feedback joint representation learning, which establishes a cross-view geolocation network with a dual-branch architecture. The basic architectures of the two branches are similar and the weights of the Swin-B network are shared. Each branch consists of three forward stages: feature extraction, feature aggregation, and feature classification. In the feature extraction and feature aggregation stages, the Swin-B network, multi-level feedback network, and adaptive region elimination strategy are used.

[0006] The UAV view images and satellite view images are respectively input into the branches of the cross-view geolocation network. After feature extraction and feature aggregation, the features are classified by a classifier. The classifier consists of a fully connected layer, a batch normalization layer, a Dropout layer, and a classification layer. The optimization process of the cross-view geolocation network is accomplished through three loss functions: cross-entropy loss function, KL divergence loss function, and feature correlation loss function.

[0007] The input to the multi-level feedback network is a feature map extracted by the swin-b network;

[0008] For a given image with different views, feature maps are extracted using the Swin-B network. This process is represented as follows:

[0009]

[0010] Where, x i For the input image, For image x i Extracted global feature map;

[0011] The extracted global feature maps are processed hierarchically, resulting in image-level features. Partial-level features and patch-level features Three levels; patch-level features in multi-level feedback networks The global receptive field is further expanded by using an MLP layer to deeply mine the potential information of local features. After processing each patch-level feature, the patch-level features are segmented according to the partition-level feature segmentation method. Combining to generate new partial-level features Patch-level features It consists of each small patch in the image-level features; the new part-level features Further combining generates a first image-level feature with the same size as the original feature. Then, it is added pixel-by-pixel to the original global features to generate the first part of the global features generated by the multi-level feedback network.

[0012] New partial-level features Compared with partial features in the original global feature map Pixel-by-pixel addition is performed, and the global receptive field is expanded through an MLP layer to generate a second image-level feature with the same size as the original feature. Second image-level features and image-level features The second part of the global features is generated by pixel-by-pixel addition.

[0013] The hierarchical processing is as follows: image-level features are complete global feature maps, partial-level features are horizontally sliced ​​based on image-level features, and patch-level features are vertically sliced ​​based on partial-level features.

[0014] The adaptive region elimination strategy first calculates the global feature map and patch-level features. The similarity matrix P is constructed based on the feature similarity between the features, and this similarity matrix is ​​represented as follows:

[0015]

[0016] Where D[·] represents the calculation of feature similarity, which uses cosine similarity to calculate the similarity between features, and k and l are image-level features. Length and width;

[0017] Based on the similarity matrix, unimportant information is removed using an adaptive region elimination mechanism; the adaptive region elimination mechanism is expressed as follows:

[0018]

[0019]

[0020] in, This indicates the updated patch-level features. Indicates the updated patch-level features The image-level global features are composed of;

[0021] First, classify the patch-level features according to feature similarity. Sort the features, starting with the patch-level features with the lowest feature similarity. Then, sort the features based on their similarity to the image-level features. The feature similarity between them is less than that between the current patch-level features and the updated image-level features. These patch-level features will be deleted until the opposite occurs, at which point the process terminates, while patch-level features that have not been evaluated will remain unchanged; the updated patch-level features will then be removed. Combining to generate new image-level features

[0022] The inputs to the classifier are the original image-level features. Part Two: Global Features Part 1: Global Features With the new image-level features, the classification process of the classifier is formalized as follows:

[0023]

[0024] in, Output vectors for the classifier module. The dimension is equal to the number of geographic label categories C.

[0025] The cross-entropy loss function is defined as follows:

[0026]

[0027]

[0028] in, It is the logit score of the real geographic label y, which is the probability score normalized by the softmax function in formula (7). for The predicted probability of belonging to the geographical label y;

[0029] The KL divergence loss function L KL The definition is as follows:

[0030]

[0031]

[0032]

[0033] Among them, O d O represents the output of the drone's view image. s The output represents the satellite image;

[0034] The feature-related loss function L sim The definition is as follows:

[0035]

[0036] Where L2[·] represents the L2 distance between features, Represents image-level features from the drone's perspective. This indicates the new image-level features after feature removal. This represents the corresponding positive sample image-level features from the satellite perspective. This represents the new image-level features of a satellite view image after feature removal. This represents the corresponding positive sample image-level features from the drone's perspective. Represents satellite-view image-level features;

[0037] L total =L cro +L KL +L sim (12).

[0038] The beneficial effects of this invention are as follows: Addressing the issue of non-robust feature extraction in previous cross-view geolocation algorithms, this invention proposes a multi-level feedback joint representation learning network to solve cross-view geolocation tasks. First, we perform segment-level and block-level operations on the extracted global features and use these features to feed back into the global features, thereby mining contextual information within the global features and improving the robustness of feature extraction. Furthermore, since the image content acquired from different platforms varies greatly and contains much interference, we introduce an adaptive region elimination strategy to remove interference from the global information, assisting the model in acquiring key information. Based on this, we introduce a feature-related loss function to assist the model in learning the features after eliminating interference, thereby improving model performance. Finally, testing on the University-1652 dataset demonstrates that the proposed network model outperforms state-of-the-art algorithms, validating the algorithm's effectiveness. Attached Figure Description

[0039] Figure 1 This is a flowchart of a cross-view geolocation method based on multi-level feedback joint representation learning. Detailed Implementation

[0040] Figure 1 This is the main flowchart of the technical solution of this invention. For example... Figure 1 As shown, the cross-view geolocation algorithm based on multi-level feedback joint representation learning proposed in this invention includes the following steps:

[0041] Step 1, Training Data: This invention constructs a relevant training set using RGB images from the University-1652 dataset and provides input images for the network;

[0042] Step 2, Model Training: The network framework design proposed in this invention uses Swin-B as the backbone network and adds a new classifier. This classifier consists of fully connected layers, batch normalization layers, Dropout layers, and classification layers. The entire network has a dual-branch architecture, with similar basic architectures and shared backbone network weights. Each branch consists of three forward stages: feature extraction, feature aggregation, and feature classification. Furthermore, in the feature extraction and feature aggregation stages, we introduce multi-level feedback and adaptive similarity feature elimination strategies to further enhance model performance. The model optimization process is accomplished through three loss functions: cross-entropy loss function, KL divergence loss function, and feature correlation loss function. These will be described in detail below.

[0043] a. Multi-level feedback network

[0044] To better extract local information from images acquired from different platforms, we designed a multi-level feedback mechanism to enhance the ability to mine contextual information. Specifically, this mechanism further mines local information from the image by using a local-to-global feedback approach, enabling the network model to focus more on the key information of the image.

[0045] For a given image from a different platform, feature maps are extracted using a backbone network. This process can be represented as follows:

[0046]

[0047] Where, x i For the input image, For image x i Extracted global feature map.

[0048] Because the transformer has a strong ability to extract local information, we retain the extracted global feature map as part of the joint representation learning to further improve the model's performance. Furthermore, due to the significant differences in image perspectives across different platforms, simply using the extracted global features does not yield ideal performance. Therefore, we adopt a multi-level feedback approach to deeply mine the latent information in the image. First, we perform hierarchical processing on the extracted global feature map, i.e., image-level feature processing. Partial-level features and patch-level features Three levels. Because images acquired from different platforms will have varying content due to changes in viewpoint, other information within the image is also important. Therefore, we start with patch-level features. We begin by expanding the global receptive field further through an MLP (Multilayer Perceptron) layer to deeply mine the latent information of local features. After processing each block, we then segment the patch-level features according to the partition-level feature segmentation method. Combining to generate new partial-level features Patch-level features It consists of small blocks of image-level features. These blocks contain a lot of information that is ignored by the global features, and this information is combined into new part-level features after further mining. Therefore, we will introduce new partial-level features. Further combining generates image-level features with the same size as the original features. Then, it is added pixel-by-pixel to the original global features to generate the first part of the global features provided by the multi-level feedback mechanism for joint representation learning. This feature incorporates patch-level features that have undergone global receptive field expansion processing. We will pay more attention to local information in the image. Therefore, we hope that this part of the feature can help the model to extract as much local information as possible from the image and further improve the model's performance.

[0049] Building upon this, we aim for the model to not only focus on local information but also on the contextual information within the image. Therefore, the newly generated partial-level features... Compared with the partial features in the original global features Pixel-by-pixel summation is performed, and the global receptive field is expanded using an MLP (Multilayer Perceptron) layer to capture as much key information as possible. After these operations, the processed block-level features are combined to generate image-level features with the same size as the original features. Finally, we will use image-level features and image-level features Pixel-wise addition is performed to generate the second part of global features for joint representation learning. Clearly, after feedback processing of block-level and partial-level features, the feature incorporates key information, which will help the newly generated feature pay more attention to the contextual information in the image.

[0050] The multi-level feedback mechanism mainly provides three types of features for the joint representation learning part: the original image-level features that focus on global information. The second part of the contextual information focuses on global features. And the first part of focusing on local information, global features These features will be input into Figure 1 The classifier shown can be formalized as follows:

[0051]

[0052] in, Output vectors for the classifier module. The dimension is equal to the number of geographic label categories C.

[0053] b. Adaptive Region Elimination Strategy

[0054] Because images acquired through different methods have different perspectives, the content within the images can also be skewed. Besides some unrecognizable changes in appearance due to significant perspective shifts, there may also be irrelevant content or unimportant local information. This information can interfere with the model's results, thus reducing its performance. Therefore, we designed an adaptive region elimination strategy to remove interfering information from the images, allowing the model to focus more on the key information within the image.

[0055] To eliminate interfering information in the features, we first construct a similarity matrix P by calculating the feature similarity between global features and local features. This similarity matrix is ​​represented as follows:

[0056]

[0057] Where D[·] represents the calculation of feature similarity, here we use cosine similarity to calculate the similarity between features, and k and l are image-level features. The length and width.

[0058] After determining the similarity relationship between local and global features, unimportant information needs to be removed. However, removing too much information may delete effective information from the features, affecting their comprehensiveness, while removing too little information will result in features containing a lot of invalid information, making it impossible to focus on the key information. Therefore, how to remove unimportant information is a crucial issue. To address this, we designed an adaptive elimination mechanism to determine the number of block-level features that need to be eliminated. This mechanism can be represented as follows:

[0059]

[0060]

[0061] in, This indicates the updated patch-level features. Indicates the updated patch-level features The resulting image-level global features.

[0062] Specifically, we determine the amount of information to be removed by continuously comparing the updated image-level features and patch-level features. In this process, we first compare the patch-level features based on feature similarity. Sort the features, starting with the patch-level features with the lowest feature similarity. Then, sort the features based on their similarity to the image-level features. The feature similarity between them is less than that between the current patch-level features and the updated image-level features. These features will be deleted until the opposite occurs, at which point the process terminates, while block-level features that have not been evaluated will remain unchanged. At this point, the updated patch-level features will be... Combining to generate new image-level features

[0063] Finally, the newly generated image-level features This will serve as the final part of the joint representation learning process. This feature section eliminates interfering information from the features and focuses more on the key information in the image, thereby further improving the model's performance. Clearly, this feature will also be input into... Figure 1 The classifier shown is used for training the network model.

[0064] c. Loss Function

[0065] To train the proposed multi-level feedback joint representation learning network model, we employ a set of hybrid loss functions, including the cross-entropy loss function L. cro KL divergence loss function L KL Feature-related loss function L sim The overall loss function can be expressed as follows:

[0066] L total =L cro +L KL +L sim (6)

[0067] First, we choose to introduce the cross-entropy loss function to assist in the training of the network model. The cross-entropy loss function is defined as follows:

[0068]

[0069]

[0070] in, It is the logit score of the real geographic label y, which is the probability score normalized by the softmax function in formula (8). for The predicted probability of belonging to the geographical label y.

[0071] Furthermore, since cross-view geolocation is considered a multi-input, multi-output task, we introduce the KL divergence loss function L. KL This is used to establish learning relationships between outputs from different domains, thereby reducing the distance between similar instances. The KL divergence loss function is used.KL The definition is as follows:

[0072]

[0073]

[0074]

[0075] Among them, O d O represents the output of the drone's view image. s This indicates the output of the satellite image.

[0076] Finally, since we designed an adaptive similarity feature elimination strategy in the network, we designed a feature correlation loss function to constrain the amount of information eliminated by the adaptive feature elimination, thereby enabling the network model to achieve better results. Feature correlation loss function L sim The definition is as follows:

[0077]

[0078] Where L2[·] represents the L2 distance between features, Represents image-level features from the drone's perspective. This represents the image-level features after feature removal. This represents the corresponding positive sample image-level features from the satellite perspective.

[0079] Step 3, Image Retrieval: In the testing phase, we first input the library images into the corresponding branch network to obtain features, and then use the classifier module to obtain the final features. These features are then concatenated and stored as a query library. Based on this, the image to be queried is input into the corresponding branch to obtain the final image. The obtained features are then compared with the features in the query library to find the geographic target closest to the query image from images on different platforms, thus completing the retrieval task.

[0080] To verify the effectiveness of the algorithm in long-term visual positioning accuracy, this invention was tested on the University-1652 dataset. According to the experimental results, the proposed algorithm achieves Recall@1 and AP of 91.87% and 93.15% respectively when retrieving satellite-view imagery from UAV-view imagery, and 95.29% and 91.51% respectively when retrieving UAV-view imagery from satellite-view imagery.

Claims

1. A cross-view geolocation method based on multi-level feedback joint representation learning, characterized in that, A cross-view geolocation network is established, which has a dual-branch architecture. The two branches have similar basic architectures and share the weights of the Swin-B network. Each branch consists of three forward stages: feature extraction, feature aggregation, and feature classification. In the feature extraction and feature aggregation stages, the method is achieved through the swin-b network, multi-level feedback network, and adaptive region elimination strategy. The UAV view images and satellite view images are respectively input into the various branches of the cross-view geolocation network. After feature extraction and feature aggregation, the features are classified by a classifier. The classifier consists of a fully connected layer, a batch normalization layer, a Dropout layer, and a classification layer. The optimization process of the cross-view geolocation network is accomplished through three loss functions: cross-entropy loss function, KL divergence loss function, and feature correlation loss function. The input of the multi-level feedback network is the feature map extracted by the Swin-B network. For a given image with different views, feature maps are extracted using the Swin-B network. This process is represented as follows: （1） in, For the input image, For image Extracted global feature map; The extracted global feature maps are processed hierarchically, resulting in image-level features. Partial-level features and patch-level features Three levels; patch-level features in multi-level feedback networks The global receptive field is further expanded by using an MLP layer to deeply mine the potential information of local features. After processing each patch-level feature, the patch-level features are segmented according to the partition-level feature segmentation method. Combining to generate new partial-level features Patch-level features It consists of each small patch in the image-level features; the new part-level features Further combining generates a first image-level feature with the same size as the original feature. This is then added pixel-by-pixel to the original global features to generate the first part of the global features generated by the multi-level feedback network. ; New partial-level features Compared with partial features in the original global feature map Pixel-by-pixel addition is performed, and the global receptive field is expanded through an MLP layer to generate a second image-level feature with the same size as the original feature. ; to the second image-level features and image-level features The second part of the global features is generated by pixel-by-pixel addition. ; The hierarchical processing is as follows: image-level features are complete global feature maps, partial-level features are horizontally sliced ​​based on image-level features, and patch-level features are vertically sliced ​​based on partial-level features.

2. The cross-view geolocation method based on multi-level feedback joint representation learning according to claim 1, characterized in that, The adaptive region elimination strategy first calculates the global feature map and patch-level features. The similarity matrix P is constructed based on the feature similarity between the features, and this similarity matrix is ​​represented as follows: （2） in, To calculate feature similarity, cosine similarity is used to determine the similarity between features, where k and l are image-level features. Length and width; Based on the similarity matrix, unimportant information is removed using an adaptive region elimination mechanism; the adaptive region elimination mechanism is expressed as follows: （3） （4） in, This indicates the updated patch-level features. Indicates the updated patch-level features The image-level global features are composed of; First, classify the patch-level features according to feature similarity. Sort the features, starting with the patch-level features with the lowest feature similarity. Then, sort the features based on their similarity to the image-level features. The feature similarity between them is less than that between the current patch-level features and the updated image-level features. These patch-level features will be deleted until the opposite occurs, at which point the process terminates, while patch-level features that have not been evaluated will remain unchanged; the updated patch-level features will then be... Combining to generate new image-level features .

3. The cross-view geolocation method based on multi-level feedback joint representation learning according to claim 2, characterized in that, The inputs to the classifier are the original image-level features. Part Two: Global Features Part 1 Global Features With the new image-level features, the classification process of the classifier is formalized as follows: （5） in, Output vectors for the classifier module. The dimension is equal to the number of geographic label categories C.

4. The cross-view geolocation method based on multi-level feedback joint representation learning according to claim 3, characterized in that, The cross-entropy loss function is defined as follows: （6） （7） in, It is the logit score of the real geographic label y, which is the probability score normalized by the softmax function in formula (7). for The predicted probability of belonging to the geographical label y; The KL divergence loss function The definition is as follows: （8） （9） （10） in, This represents the output of the drone's view image. The output represents the satellite image; The feature-related loss function The definition is as follows: （11） in, Indicates the L2 distance between features. Represents image-level features from the drone's perspective. This represents the new image-level features after feature removal from the drone's perspective. This represents the corresponding positive sample image-level features from the satellite perspective. This represents the new image-level features of a satellite view image after feature removal. This represents the corresponding positive sample image-level features from the drone's perspective. Represents satellite-view image-level features; （12）。

Images