A cross-view geolocation method based on feature center region diffusion and edge radiation

Abstract

This invention belongs to the field of computer vision technology, and specifically relates to a cross-view geolocation method based on feature center region diffusion and edge radiation. A key feature center region diffusion module is proposed to further focus on key regions in the image, thereby improving the discriminative power of the extracted features. Building upon this, to prevent overlooking some effective information in the surrounding area while searching for key regions, an edge radiation mechanism is designed to further mine effective information in the image. Furthermore, this invention introduces a symmetric triple INFONCE loss function to assist the model in acquiring robust global and local features. Finally, tests were conducted on two cross-view geolocation datasets, University-1652 and SUES-200. Experimental results demonstrate that the proposed network model outperforms state-of-the-art algorithms, proving the effectiveness of the method.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a cross-view geolocation method based on feature center region diffusion and edge radiation. Background Technology

[0002] The core of cross-view geolocalization is to estimate the location information of a target by matching images from different viewpoints or different acquisition sources. It has potential applications in many fields, such as drone navigation, agricultural robots, target detection, and autonomous driving. Although these tasks can be achieved using real-time kinematics and GPS, these sensors are expensive and susceptible to short-term signal interruptions that can affect the final results. Especially in urban areas, the urban canyon effect can introduce greater biases into the localization results. Aerial image-based localization can further improve accuracy, and cross-view geolocalization is an effective way to address these problems. Therefore, research on cross-view geolocalization methods is of great significance.

[0003] The paper "IEEE / CVF International Conference on Computer Vision, 16847-16856, 2023" obtains robust global features through a contrastive learning architecture based on symmetric InfoNCE loss and introduces a new sampling strategy to improve model performance. However, it only describes images from different perspectives using global features, but images also contain some local information, which is also key to solving the problem.

[0004] The paper "IEEE Transactions on Geoscience and Remote Sensing, 1–11, 2023" utilizes a global self-attention mechanism for segmentation and fusion to fully mine features, and introduces target feature enhancement to obtain discriminative features, thereby completing cross-view geolocation tasks. However, it introduces too many modules, which, while improving model performance, also increase computational complexity, resulting in suboptimal model performance. Summary of the Invention

[0005] The purpose of this invention is to propose a cross-view geolocation method based on feature center region diffusion and edge radiation, thereby improving the accuracy and robustness of cross-view geolocation.

[0006] The technical solution of the present invention is as follows: A cross-view geolocation method based on feature center region diffusion and edge radiation, comprising the following steps:

[0007] Images from different platform perspectives are acquired and input into the backbone network; the backbone network has a dual-branch architecture, with each branch having the same structure and sharing the backbone network weights; each branch structure includes a backbone network, a key feature center region diffusion module, an edge radiation mechanism, and a classifier module;

[0008] The backbone network is based on ConvNeXt-base; images from different platform perspectives are input into the backbone network to extract global features of the image;

[0009] The global features are respectively input to the classifier module and the key feature center region diffusion module;

[0010] The global features, the features obtained by the central region diffusion module of key features, and the features obtained by the edge radiation mechanism are added together to form a new feature with the same size as the original global features. The new feature is then fed into the classifier module after passing through an MLP layer.

[0011] The overall backbone network is optimized based on the loss function, and the optimized backbone network is used to solve the cross-view geolocation problem.

[0012] During the testing phase, the reference image is first input into the corresponding branch network to obtain features, and the final features are generated by the classifier. These features are stored as a reference library for the query. Based on this, the features to be queried are input into another branch network to obtain the features of the input image. The features of the input image are compared with the features in the reference query library to find the image of the same scene that is closest to the query image from different perspectives, thus completing the cross-view geolocation task.

[0013] The process of extracting global features from the backbone network is represented as follows:

[0014]

[0015] Where, x i For the input image, For the input image x i Extracted global features.

[0016] The key feature center region diffusion module is specifically as follows:

[0017] The key feature center region diffusion module extracts key features from the global features and uses these key features as the center to further diffuse the key features.

[0018] The key feature center region diffusion module uses two relationships—feature heatmap and feature similarity—to determine which n key features are selected as the center region.

[0019] Based on two relationships—feature heatmap and feature similarity—a heatmap judgment matrix and a similarity judgment matrix are obtained respectively. These two matrices are then added together and sorted from highest to lowest to select initial key feature center regions. During the selection process, features within adjacent nine-square grids that serve as center regions are deleted, and features from other regions with high summation values ​​are used to supplement them, thus determining the final key feature centers. After determining these key feature centers, the diffusion proceeds outwards from these center regions. During this diffusion process, all center regions simultaneously diffuse outwards in a nine-square grid manner, stopping when the diffusion range of a feature overlaps with or touches the boundary of other feature diffusion ranges. Finally, the diffused features are combined to form a component of the local information section.

[0020] The heatmap judgment matrix is ​​obtained as follows:

[0021] The feature vectors obtained from global features using the CBAM algorithm are averaged based on channels to represent the heat value of the feature.

[0022]

[0023] Among them, t i,j P represents the heat value in the ith-th row and j-th column of the heatmap judgment matrix. k This represents the feature heat value of the kth channel of the feature vector corresponding to the current part, where m and n are the length and width of the generated matrix; H is the heatmap judgment matrix; and S is the number of channels of the feature vector.

[0024] The similarity judgment matrix is ​​obtained as follows: Global features are averaged based on channels and then divided into multiple small blocks to form local features. The cosine similarity between the global features and each local feature is calculated to form a similarity judgment matrix A.

[0025]

[0026] Where D[·] represents the process of calculating feature similarity, using cosine similarity to calculate the similarity between global and local features, and m and n are the length and width of the generated matrix.

[0027] The edge radiation mechanism processes the features obtained by the diffusion module in the central region of the key features. The specific process is as follows:

[0028] After the key feature center region diffusion module determines the feature center region, these key features are aggregated and their boundaries are determined. Since the aggregated features are not of a relatively regular size, the boundary range of the features is determined based on the current irregular region and expanded to the boundaries of each edge of the irregular image, thereby expanding the range of the entire feature selection area and generating edge radiation features.

[0029] A square ring partitioning strategy is used to divide the edge radiation features into regions and generate local features; the square ring partitioning strategy divides the entire edge radiation features into blocks according to the distance from the image center.

[0030] The loss function is divided into the cross-entropy loss function L. cro and adaptive triple INFONCE loss function L tri_Info The loss function is expressed as follows:

[0031] L total =L cro +L tri_Info (6)

[0032] The cross-entropy loss function L cro The definition is as follows:

[0033]

[0034] in, It is the logit score of the real geographic label y. for The predicted probability of belonging to the geographical label y;

[0035] The adaptive triple INFONCE loss function is used to constrain the backbone network; the adaptive triple INFONCE loss function is defined as follows:

[0036]

[0037] Where q represents the encoded query feature, and R is a set of reference encoded features; r + To query positive samples of feature q, r - To find negative samples of feature q; τ is a hyperparameter, a static value, m f For margin, α f These are the weight parameters.

[0038] The images from different platform perspectives include satellite imagery and UAV perspective images, and the adaptive triple INFONCE loss function L... tri_Info It is expressed as follows:

[0039]

[0040] The adaptive triple INFONCE loss function for satellite imagery. This represents the adaptive triple INFONCE loss function for images viewed from the drone's perspective.

[0041] The classifier module consists of a fully connected layer, a batch normalization layer, a Dropout layer, and a classification layer.

[0042] The beneficial effects of this invention: Addressing the difficulty in extracting robust features in previous cross-view geolocation algorithms, this invention proposes a cross-view geolocation method based on feature center region diffusion and edge radiation to solve cross-view geolocation tasks. Specifically, this invention employs a joint representation learning approach using global and local information to further improve model performance. First, this invention recognizes that focusing solely on global information in the image is insufficient. Therefore, it proposes a key feature center region diffusion module to further focus on key regions in the image, thereby improving the discriminative power of the extracted features. Furthermore, to prevent overlooking surrounding effective information when searching for key regions, an edge radiation mechanism is designed to further mine effective information in the image. In addition, this invention introduces a symmetric triple INFONCE loss function to assist the model in acquiring robust global and local features. Finally, tests were conducted on two cross-view geolocation datasets, University-1652 and SUES-200. Experimental results demonstrate that the proposed network model outperforms state-of-the-art algorithms, proving the effectiveness of the method. This invention effectively extracts key features from images from different perspectives by using a diffusion module based on the central region of key features and an edge radiation mechanism. This increases the robustness of the model in extracting features, enabling the network to effectively distinguish images of the same scene from different perspectives and better complete cross-perspective geographic tasks. Attached Figure Description

[0043] Figure 1 This is a flowchart of a cross-view geolocation method based on feature center region diffusion and edge radiation. Detailed Implementation

[0044] Figure 1 This is the main flowchart of the technical solution of this invention. For example... Figure 1 As shown, the cross-view geolocation method based on feature center region diffusion and edge radiation proposed in this invention includes the following steps:

[0045] (1) Training data: This invention constructs a relevant training set using RGB images from the University-1652 dataset to provide input images;

[0046] (2) Model training: The backbone network proposed in this invention is a dual-branch architecture, including that each branch has the same structure and the backbone network weights are shared; each branch structure includes a backbone network, a key feature center region diffusion module, an edge radiation mechanism and a classifier module;

[0047] The backbone network is based on ConvNeXt-base; images from different platform perspectives are input into the backbone network to extract global features of the image;

[0048] The classifier module consists of a fully connected layer, a batch normalization layer, a Dropout layer, and a classification layer.

[0049] Furthermore, in order to better extract key information from images, the key feature center region diffusion module and edge radiation mechanism proposed in this invention assist the model in further extracting key information from images.

[0050] The global features are respectively input to the classifier module and the key feature center region diffusion module;

[0051] The global features, the features obtained by the central region diffusion module of key features, and the features obtained by the edge radiation mechanism are added together to form a new feature with the same size as the original global features. The new feature is then fed into the classifier module after passing through an MLP layer.

[0052] The optimization process of the backbone network is accomplished through two loss functions: the cross-entropy loss function and the symmetric-based adaptive triple InfoNCE loss function.

[0053] The present invention will now describe these in detail.

[0054] a. Diffusion of key features in the central region

[0055] While global features can achieve some success in cross-view geolocation tasks, the significant differences in image content across different viewpoints make it difficult to achieve ideal results. To obtain more robust features, this invention proposes a key feature region center diffusion module to assist the model in extracting as much key information as possible from the image. Specifically, this module extracts key parts from global features and uses these extracted key features as centers to further diffuse these features, thereby making the module more focused on the key information within the global features.

[0056] After obtaining the global features of the image, this invention retains this portion of the global feature map as part of the joint representation learning to serve the final model, thereby improving the overall performance of the proposed network model. Since the information about the same scene contained in images from different perspectives varies greatly depending on the content of the images, this invention introduces local information into the backbone network model training to improve the feature discrimination of the final network model together with the global features, in order to better ensure the model's ability to distinguish the same scene. To this end, this invention proposes a key feature center region diffusion module. This module expands the region of key features by acquiring key regions in the image features and diffusing them outwards from these central local regions, allowing the proposed network model to pay more attention to key information in the image and expand the network's receptive field.

[0057] Specifically, after acquiring the global features of the input image, this invention inputs these global features into the key feature region center diffusion module. Therefore, determining the selected feature center region becomes a challenge. To better determine the selection area for the key feature center, this invention uses two relationships: feature heatmap and feature similarity, to determine which n features are selected as the center region. The image heatmap clearly shows that the main building has a higher heat value, while other background areas have lower heat values. This also indicates that the model pays different attention to different regions in the image, resulting in certain patterns in the heatmap distribution. Therefore, this invention chooses the feature heatmap method as one of the evaluation criteria. The CBAM algorithm deeply mines key parts of image features through channel attention and spatial attention, and has been widely applied in many fields. Therefore, this invention chooses the CBAM algorithm to extract key regions of image features as one of the evaluation criteria. After acquiring the global features, these features are input into CBAM to obtain the heat value of each part. This process can be formulated as follows:

[0058]

[0059] Where t i,j P represents the heat value in the ith-th row and j-th column of the heatmap judgment matrix. k This represents the feature heat value of the kth channel of the feature vector corresponding to the current part, where m and n are the length and width of the generated matrix. In short, this invention performs a channel-based averaging operation on the acquired feature vector to represent the heat value of that feature. In this way, this invention can obtain a heatmap of the input features and use it to assist the module in selecting the feature center region.

[0060] Furthermore, since heatmap-based methods may be unstable in some scenarios, some key edge regions in the image may not be fully detected. To address this, this invention designs a feature similarity-based method to assist in selecting feature points for key regions. In the feature similarity-based part, to find the required feature centers, this invention divides the global features into multiple small blocks after averaging based on channels to form local features. Based on this, the cosine similarity between the global features and each local feature is calculated to construct a feature similarity matrix A, which can be represented as follows:

[0061]

[0062] Where D[·] represents the process of calculating feature similarity. Here, the present invention uses cosine similarity to calculate the similarity between global features and local features, and m and n are the length and width of the generated matrix.

[0063] After obtaining these two sets of judgment information, this invention adds the heatmap judgment matrix and the similarity judgment matrix and sorts them from high to low to select the initial key feature center regions. Since this invention expands outwards after selecting the center, thus increasing the range of the key region, to prevent the center regions from being too densely selected, this invention deletes features within adjacent nine-square grids that are considered center regions during the selection process and supplements them with other region features with higher values, thereby determining the final key feature centers. After determining these feature centers, this invention expands outwards based on these center regions. During the expansion process, all center regions simultaneously expand outwards in a nine-square grid manner. Expansion stops when the feature expansion range overlaps with or touches the boundary of other feature expansion ranges. Finally, the expanded features are combined to form a component of the local information part for subsequent operations.

[0064] b. Edge radiation mechanism

[0065] The key feature center region diffusion module allows the model to focus more on key information in the image, thereby improving the model's feature discrimination of the same scene from different viewpoints. However, after diffusion based on the key feature center region, the diffused area may become too concentrated, resulting in the key area of ​​focus being too small, which can lead to a decrease in model performance. To address this, this invention designs an edge radiation mechanism to help the model better focus on key information in the image, facilitating subsequent operations on local features.

[0066] Specifically, this invention first determines the selected feature region based on the features diffused by the key feature center region diffusion module. However, this selected region may neglect important surrounding information in order to focus on key information in the image. Therefore, in order to expand the receptive field of the model while focusing on more key information, this invention uses a regional feature radiation mechanism to increase the feature region of interest, thereby assisting local features to improve the performance of the final network model. After the feature region is determined by the feature center region diffusion module, this invention aggregates these features and determines their boundaries. To expand the receptive field of these features, this invention expands the current region to the boundaries of each edge of the irregular image, thereby expanding the entire feature selection area, and incorporates these features as part of the local features for subsequent operations.

[0067] Building upon this foundation, to demonstrate the local features required for generating contextual information, this invention employs a square ring partitioning strategy to divide the processed features into regions, thereby generating local and global features for joint representation learning. The center of the partitioned image is approximately aligned with the center of the feature map; therefore, the square ring partitioning strategy divides the entire image into blocks based on their distance from the image center. Although image content varies significantly across different viewpoints, the regions in the cross-view image are approximately spatially aligned after partitioning. This also increases the feature similarity among the partitioned parts, thereby improving the performance of the final model. Therefore, this invention selects these features as both local and global features to constitute the final model features for effectively completing cross-view geolocation tasks.

[0068] After determining these two parts of features, the original global features, the features generated by the diffusion module in the central region of the key features, and the features generated by the edge radiation mechanism are added together to form a new feature with the same size as the original global features. The newly generated feature is then passed through an MLP layer to further expand the receptive field of the feature.

[0069] c. Loss Function

[0070] To better train the proposed joint representation learning network based on feature center region diffusion and aggregation, this invention employs a set of hybrid loss functions, including two parts: the cross-entropy loss function L. cro and adaptive triple InfoNCE loss function L tri_Info The overall loss function for training the network can be expressed as follows:

[0071] L total =L cro +L tri_Info (5)

[0072] Cross-entropy loss function is widely used in classification tasks. Therefore, it is introduced to assist in model training. The cross-entropy loss function L... cro The definition is as follows:

[0073]

[0074] in, It is the logit score of the real geographic label y. for The predicted probability of belonging to the geographical label y.

[0075] Furthermore, to better assist the model in obtaining discriminative features, this invention designs an adaptive triple INFONCE loss function to constrain the network model, thereby enabling the model to achieve better performance. The adaptive triple INFONCE loss function can be defined as follows:

[0076]

[0077] Where q represents the encoded query feature, and R is a set of reference encoded features. + To query positive samples of feature q, r - To find negative samples for the query feature q. τ is a hyperparameter that can be learned or set to a static value. Additionally, m... f For margin, α f These are the weight parameters.

[0078] To address this, the overall adaptive triple INFONCE loss function L for training the network model is used. tri_Info It can be represented as follows:

[0079]

[0080] d. Image retrieval

[0081] Since the proposed network model is a two-branch network structure, during the testing phase, the reference image is first input into the corresponding branch network to obtain features, and then the final features are generated by a classifier. These features are then stored as a reference library for queries. Based on this, the features to be queried are input into the other branch network to obtain the features of the input image. Afterwards, the features of the input image are compared with the features in the reference query library, thereby finding images of the same scene from different perspectives that are closest to the query image in the query library, thus completing the cross-view geolocation task.

[0082] To verify the effectiveness of the algorithm in cross-view geolocation accuracy, this invention was tested on the University-1652 dataset. According to the experimental results, the proposed algorithm achieves Recall@1 and AP of 92.79% and 93.91% respectively when retrieving satellite view imagery from UAV view imagery, and 95.58% and 92.17% respectively when retrieving UAV view imagery from satellite view imagery.

Claims

1. A cross-view geolocation method based on feature center region diffusion and edge radiation, characterized in that, The steps include the following: Images from different platform perspectives are acquired and input into the backbone network; the backbone network has a dual-branch architecture, with each branch having the same structure and sharing the backbone network weights; each branch structure includes a backbone network, a key feature center region diffusion module, an edge radiation mechanism, and a classifier module; The backbone network is based on ConvNeXt-base; images from different platform perspectives are input into the backbone network to extract global features of the image; The global features are respectively input to the classifier module and the key feature center region diffusion module; The global features, the features obtained by the central region diffusion module of key features, and the features obtained by the edge radiation mechanism are added together to form a new feature with the same size as the original global features. The new feature is then fed into the classifier module after passing through an MLP layer. The overall backbone network is optimized based on the loss function, and the optimized backbone network is used to solve the cross-view geolocation problem. During the testing phase, the reference image is first input into the corresponding branch network to obtain features and the final features are generated by the classifier. These features are stored as a reference library for the query. The features to be queried are input into the network of another branch to obtain the features of the input image. The features of the input image are compared with the features in the reference query library to find the image of the same scene that is closest to the query image from different perspectives to complete the cross-view geolocation task. The key feature center region diffusion module is specifically as follows: The key feature center region diffusion module extracts key features from the global features and uses these key features as the center to further diffuse the key features. The key feature center region diffusion module uses two relationships—feature heatmap and feature similarity—to determine which... A key feature was selected as the central region; Based on the two relationships of feature heatmap and feature similarity, heatmap judgment matrix and similarity judgment matrix are obtained respectively; the heatmap judgment matrix and similarity judgment matrix are added together and sorted from high to low to select the initial key feature center region; During the selection process, features that are considered as the central region within adjacent nine-square grids are deleted, and features from other regions with high summation values ​​are used to supplement them, thereby determining the final key feature center. After determining these key feature centers, the diffusion is spread outwards from these key feature center regions. During the diffusion process, all center regions simultaneously spread outwards in a nine-square grid manner. Diffusion stops when the diffusion range of a feature overlaps with the diffusion range of other features or touches the boundary. Finally, the diffused features are combined together to form a component of the local information part. The edge radiation mechanism processes the features obtained by the diffusion module in the central region of the key features. The specific process is as follows: After the key feature center region diffusion module determines the feature center region, these key features are aggregated and their boundaries are determined. Since the aggregated features are not of a relatively regular size, the boundary range of the features is determined based on the current irregular region and expanded to the boundaries of each edge of the irregular image, thereby expanding the range of the entire feature selection area and generating edge radiation features. A square ring partitioning strategy is used to divide the edge radiation features into regions and generate local features; the square ring partitioning strategy divides the entire edge radiation features into blocks according to the distance from the image center.

2. The cross-view geolocation method based on feature center region diffusion and edge radiation according to claim 1, characterized in that, The process of extracting global features from the backbone network is represented as follows: (1) ； in, For the input image, For the input image Extracted global features.

3. The cross-view geolocation method based on feature center region diffusion and edge radiation according to claim 2, characterized in that, The heatmap judgment matrix is ​​obtained as follows: The feature vectors obtained from global features using the CBAM algorithm are averaged based on channels to represent the heat value of the feature. （2）； （3）； in, The first element in the heatmap judgment matrix represents the... line, number The calorific value of the column, This represents the eigenvector corresponding to the current part. Characteristic thermal values ​​of each channel and represents the length and width of the generated matrix; H is the heatmap judgment matrix; The number of channels in the feature vector.

4. The cross-view geolocation method based on feature center region diffusion and edge radiation according to claim 3, characterized in that, The similarity judgment matrix is ​​obtained as follows: Global features are averaged based on channels and then divided into multiple small blocks to form local features. The cosine similarity between the global features and each local feature is calculated to form a similarity judgment matrix. : （4）； （5）； in, This describes the process of calculating feature similarity, using cosine similarity to calculate the similarity between global and local features. and Let be the length and width of the generated matrix.

5. The cross-view geolocation method based on feature center region diffusion and edge radiation according to claim 4, characterized in that, The loss function is divided into cross-entropy loss function. and adaptive triple INFONCE loss function The loss function is expressed as follows: （6）； The cross-entropy loss function The definition is as follows: （7）； （8）； in, It is the logit score of the real geographic label y. for The predicted probability of belonging to the geographical label y; The adaptive triple INFONCE loss function is used to constrain the backbone network; the adaptive triple INFONCE loss function is defined as follows: （9）； （10）； （11）； in For the encoded query features, A set of reference coding features; For query features Positive samples For query features Negative samples; It is a hyperparameter, and it has a static value. For margin. These are the weight parameters.

6. The cross-view geolocation method based on feature center region diffusion and edge radiation according to claim 5, characterized in that, The images from different platform perspectives include satellite imagery and UAV perspective images, using an adaptive triple INFONCE loss function. It is expressed as follows: （12）； The adaptive triple INFONCE loss function for satellite imagery. This represents the adaptive triple INFONCE loss function for images viewed from the drone's perspective.

7. The cross-view geolocation method based on feature center region diffusion and edge radiation according to claim 6, characterized in that, The classifier module consists of a fully connected layer, a batch normalization layer, a Dropout layer, and a classification layer.

Images