Cross-visual-angle geographic positioning method based on feature center region diffusion and edge radiation

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a cross-view geographic positioning method based on feature center region diffusion and edge radiation. A key feature-based central region diffusion module is proposed to further pay attention to a key region in the image so as to improve the discrimination of the extracted features. On the basis, in order to prevent some surrounding effective information from being neglected when the key area is searched, an edge radiation mechanism is designed to further mine the effective information in the image. In addition, a symmetric triple INFONCE loss function is introduced to assist the model in acquiring robust global features and local features. And finally, respectively carrying out tests on the two cross-view geographic positioning data sets, namely, University 1652 and SUES-200, and respectively carrying out tests on the two cross-view geographic positioning data sets, namely, the University 1652 and the SUES-200. Experimental results prove that the proposed network model is superior to the current most advanced algorithm, and the effectiveness of the method is proved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a cross-viewing angle geographic positioning method based on feature central area diffusion and edge radiation. Background Art

[0002] The core of the cross-view geolocation task is to estimate the location information of the target by matching different perspectives or different source images, and has potential applications in many fields, such as drone navigation, agricultural robots, target detection, autonomous driving, etc. Although these tasks can be achieved through real-time kinematics and GPS, these sensors are expensive and there will be short-term signal interruptions that affect the final results. Especially in cities, the urban canyon effect may bring greater deviations to the positioning results. Positioning based on aerial images can further improve the accuracy of positioning, and cross-view geolocation is an effective way to solve these problems. Therefore, it is of great significance to study cross-view geolocation methods.

[0003] The paper "IEEE / CVF International Conference on Computer Vision, 16847-16856, 2023" obtains robust global features through a contrastive learning architecture based on symmetric InfoNCE loss, and introduces a new sampling strategy to improve model performance. However, it only describes images from different perspectives through global features, but there is still some local information in the image, which is also the key to solving the problem.

[0004] The paper "IEEE Transactions on Geoscience and Remote Sensing, 1–11, 2023" fully mines features based on the segmentation and fusion of the global self-attention mechanism, and obtains discriminative features by introducing target feature enhancement to complete the cross-view geolocation task. Too many modules are introduced, which will increase the computational complexity of the model while improving the performance of the model, resulting in unsatisfactory performance of the model. Summary of the invention

[0005] The purpose of the present invention is to propose a cross-viewing geolocation method based on feature central area diffusion and edge radiation to improve the accuracy and robustness of cross-viewing geolocation.

[0006] The technical solution of the present invention is as follows: A cross-viewing geographical positioning method based on characteristic central area diffusion and edge radiation comprises the following steps:

[0007] Acquire images from different platform perspectives and input them into a backbone network; the backbone network is a dual-branch architecture, each branch has the same structure and the backbone network weights are shared; each branch structure includes a backbone network, a key feature central area diffusion module, an edge radiation mechanism and a classifier module;

[0008] The backbone network is based on ConvNeXt-base; images from different platform perspectives are input into the backbone network to extract global features of the image;

[0009] The global features are input into the classifier module and the key feature central area diffusion module respectively;

[0010] The global feature, the feature obtained by the key feature central area diffusion module and the feature obtained by the edge radiation mechanism are added together to form a new feature with the same size as the original global feature. The new feature is input into the classifier module after passing through an MLP layer.

[0011] The overall backbone network is optimized based on the loss function, and the optimized backbone network is used to solve the cross-view geolocation problem;

[0012] During the testing phase, the reference image is first input into the corresponding branch network to obtain features and the final features are generated through the classifier, and these features are stored as a reference library for queries; on this basis, the features to be queried are input into another branch of the network to obtain the features of the input image; the features of the input image are compared with the features in the reference query library, so as to find the images of the same scene at different perspectives that are closest to the query image in the query library to complete the cross-perspective geolocation task.

[0013] The backbone network extracts global features as follows:

[0014]

[0015] Among them, x i is the input image, For the input image x i Extracted global features.

[0016] The key feature central area diffusion module is specifically:

[0017] The key feature center area diffusion module extracts the key features from the global features, takes the key features as the center, and further diffuses the key features;

[0018] The key feature center area diffusion module uses the feature heat map and feature similarity to determine which n key features are selected as the center area;

[0019] Based on the relationship between the feature heat map and the feature similarity, the heat map judgment matrix and the similarity judgment matrix are obtained respectively; the heat map judgment matrix and the similarity judgment matrix are added and sorted from high to low to select the initial key feature center area; in the selection process, the features in the adjacent nine-square grid as the central area are deleted, and other regional features with high addition values ​​are used to supplement them, so as to determine the final key feature center; after determining these key feature centers, they are diffused to the surrounding areas based on these key feature center areas; in the diffusion process, all central areas are diffused to the surrounding areas in the form of a nine-square grid at the same time, and the diffusion is stopped when the diffusion range of the feature overlaps with the diffusion range of other features or touches the boundary; finally, the diffused features are combined together to form a component of the local information part.

[0020] The heat map judgment matrix is ​​obtained as follows:

[0021] The feature vector obtained by the CBAM algorithm of the global feature is averaged based on the channel to represent the heat value of the feature;

[0022]

[0023] Among them, t i,j represents the heat value of the ith row and jth column in the heat map judgment matrix, P k It represents the characteristic heat value of the kth channel of the feature vector corresponding to the current part, m and n are the length and width of the generated matrix; H is the heat map judgment matrix; S is the number of channels of the feature vector.

[0024] The similarity judgment matrix acquisition process is as follows: the global features are averaged based on the channel and then divided into multiple small blocks to form local features, and the cosine similarity between the global features and each local feature is calculated to form a similarity judgment matrix A:

[0025]

[0026] Where D[·] represents the process of calculating feature similarity. Cosine similarity is used to calculate the similarity between global features and local features. m and n are the length and width of the generated matrix.

[0027] The edge radiation mechanism processes the features obtained by the key feature central area diffusion module, and the specific process is as follows:

[0028] After the key feature center area diffusion module determines the feature center area, these key features are gathered together and the boundaries of these features are determined; since the gathered features are not of a relatively regular size, the boundary range of the features is determined based on the current irregular area and expanded to the boundaries of each side of the irregular image, thereby expanding the range of the entire feature selection area and generating edge radiation features;

[0029] A square ring division strategy is used to divide the edge radiation feature into regions to generate local features; the square ring division strategy divides the entire edge radiation feature into blocks according to the distance from the center of the image.

[0030] The loss function is divided into a cross entropy loss function L cro and the adaptive triple INFONCE loss function L tri_Info , the loss function is expressed as follows:

[0031] L total =L cro +L tri_Info (6)

[0032] The cross entropy loss function L cro The definition is as follows:

[0033]

[0034] in, is the logit score of the true geo-tag y, for The predicted probability of belonging to geographic label y;

[0035] The adaptive triple INFONCE loss function is used to constrain the backbone network; the adaptive triple INFONCE loss function is defined as follows:

[0036]

[0037] Where q is the encoded query feature, R is a set of reference encoded features; r + is the positive sample of query feature q, r - is the negative sample of the query feature q; τ is a hyperparameter, a static value, m f is margin,α f is the weight parameter.

[0038] The images from different platform perspectives include satellite images and drone perspective images, and the adaptive triple INFONCE loss function L tri_Info It is expressed as follows:

[0039]

[0040] is the adaptive triple INFONCE loss function for satellite images, Representing the adaptive triple INFONCE loss function for drone-view images.

[0041] The classifier module consists of a fully connected layer, a batch normalization layer, a Dropout layer and a classification layer.

[0042] Beneficial effects of the present invention: In view of the problems that it is difficult to extract robust features in previous cross-view geolocation algorithms, the present invention proposes a cross-view geolocation method based on feature center area diffusion and edge radiation to solve the cross-view geolocation task. Specifically, the present invention adopts the method of joint representation learning of global information and local information to further improve the performance of the model. First of all, the present invention believes that it is not enough to only focus on the global information in the image. In this regard, it is proposed to further focus on the key areas in the image based on the key feature center area diffusion module to improve the discriminability of the extracted features. On this basis, in order to prevent the neglect of some effective information around when searching for key areas, an edge radiation mechanism is designed to further mine the effective information in the image. In addition, the present invention also introduces a symmetric triple INFONCE loss function to assist the model in obtaining robust global and local features. Finally, tests were carried out on two cross-view geolocation datasets, namely University-1652 and SUES-200. The experimental results show that the proposed network model is superior to the most advanced algorithm and proves the effectiveness of the method. The present invention can effectively extract key features in images of different perspectives based on the key feature central area diffusion module and edge radiation mechanism, thereby increasing the robustness of the model's feature extraction, enabling the network to effectively distinguish images of the same scene at different perspectives and better complete cross-perspective geographic tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The flowchart of a cross-view geolocation method based on feature center area diffusion and edge radiation. DETAILED DESCRIPTION

[0044] Figure 1 It is the main flow chart of the technical solution of the present invention. Figure 1 As shown, the cross-viewing geographical positioning method based on characteristic central area diffusion and edge radiation proposed by the present invention includes the following steps:

[0045] (1) Training data: The present invention constructs a relevant training set using RGB images in the dataset University-1652 and provides input images;

[0046] (2) Model training: The backbone network proposed in the present invention is a dual-branch architecture, including that each branch has the same structure and the backbone network weight is shared; each branch structure includes a backbone network, a key feature central area diffusion module, an edge radiation mechanism and a classifier module;

[0047] The backbone network is based on ConvNeXt-base; images from different platform perspectives are input into the backbone network to extract global features of the image;

[0048] The classifier module consists of a fully connected layer, a batch normalization layer, a Dropout layer, and a classification layer.

[0049] In addition, in order to better mine the key information in the image, the key feature central area diffusion module and edge radiation mechanism proposed in the present invention are used to assist the model to further mine the key information in the image.

[0050] The global features are input into the classifier module and the key feature central area diffusion module respectively;

[0051] The global feature, the feature obtained by the key feature central area diffusion module and the feature obtained by the edge radiation mechanism are added together to form a new feature with the same size as the original global feature. The new feature is input into the classifier module after passing through an MLP layer.

[0052] The optimization process of the backbone network is completed through two loss functions, namely the cross entropy loss function and the symmetry-based adaptive triple InfoNCE loss function.

[0053] Next, the present invention will introduce these in detail.

[0054] a. Diffusion of key feature center area

[0055] Although global features can achieve some results in cross-view geolocation tasks, it is difficult to achieve ideal results due to the huge difference in image content under different viewpoints. In order to obtain more robust features, the present invention proposes a key feature region center diffusion module to assist the model in mining key information in the image as much as possible. Specifically, the module extracts the key parts of the global features and uses these extracted key features as the center to further diffuse these features, so that the module pays more attention to the key information in the global features.

[0056] After obtaining the global features of the image, the present invention retains this part of the global feature map as a part of the joint representation learning to serve the final model, so as to improve the overall performance of the proposed network model. Since in the cross-view geolocation task, the same scene information contained in images of different views is greatly affected by the image content of the view, in order to better ensure the model's ability to identify the same scene, the present invention introduces local information into the backbone network model training together with the global features to improve the feature identification of the final network model. In this regard, the present invention proposes a diffusion module based on the central area of ​​key features. This module expands the regional range of key features by obtaining the key areas in the image features and diffusing them around the central local features, so that the proposed network model can pay more attention to the key information in the image and expand the network's receptive field.

[0057] Specifically, after obtaining the global features of the input image, the present invention inputs this part of the global features into the key feature area center diffusion module. Therefore, how to determine the selected feature center area becomes a difficult problem. In this regard, in order to better determine the selected area of ​​the key feature center, the present invention uses two relationships, feature heat map and feature similarity, to determine which n features are selected as the center area. The image heat map can intuitively see that the main building has a larger thermal value, while other background areas have smaller thermal values. This also shows that the model pays different attention to different areas in the image, and will produce certain rules in the distribution of the heat map. Therefore, the present invention selects the feature heat map as one of the judgment bases. The CBAM algorithm deeply mines the key parts of the image features through channel attention and spatial attention, and has been widely used in many fields. Therefore, the present invention selects the CBAM algorithm to extract the key areas of the image features as one of the judgment bases. After obtaining the global features, the features are input into CBAM to obtain the thermal value of each part. The process can be formulated as follows:

[0058]

[0059] where t i,j represents the heat value of the ith row and jth column in the heat map judgment matrix, P k The characteristic heat value of the kth channel of the feature vector corresponding to the current part is represented, and m and n are the length and width of the generated matrix. In short, the present invention performs a channel-based average operation on the acquired feature vector to represent the heat value of the feature. In this way, the present invention can obtain the heat map of the input feature and use it to assist the module in selecting the feature center area.

[0060] In addition, since the heat map-based method may not be stable in some scenarios, some edge key areas in the image may not be fully noticed. In this regard, the present invention designs a method based on judging feature similarity to assist in selecting feature points in key areas. In the feature similarity-based part, in order to find the required feature center, the present invention divides the global feature into multiple small blocks based on the channel average operation to form local features. On this basis, the cosine similarity between the global feature and each local feature is calculated to form a feature similarity matrix A, which can be expressed as follows:

[0061]

[0062] Wherein, D[·] represents the process of calculating feature similarity. Here, the present invention uses cosine similarity to calculate the similarity between global features and local features, and m and n are the length and width of the generated matrix.

[0063] After obtaining these two parts of judgment information, the present invention adds the heat map judgment matrix and the similarity judgment matrix and sorts them from high to low to select the initial key feature center area. Since the present invention will spread to the surrounding areas after selecting the center to increase the range of the key area, in order to prevent the central area from being selected too densely, the present invention will delete the features of the central area in the adjacent nine-square grid during the selection process, and supplement them with other high-value regional features to determine the final key feature center. After determining these feature centers, the present invention will spread to the surrounding areas based on these central areas. In the process of further diffusion, all central areas diffuse to the surrounding areas in the form of a nine-square grid at the same time, and stop diffusing when the feature diffusion range overlaps with other feature diffusion ranges or touches the boundary. Finally, the diffused features are combined together to form a component of the local information part for subsequent operations.

[0064] b. Edge radiation mechanism

[0065] The diffusion module based on the key feature center area can make the model pay more attention to the key information in the image, thereby improving the model's feature identification of the same scene under different viewing angles. However, after diffusion based on the key feature center area, the diffused area may be too concentrated, resulting in the key area of ​​concern being too small, which leads to a decrease in model performance. In this regard, the present invention designs an edge radiation mechanism to assist the model to better pay attention to the key information in the image, so as to facilitate the subsequent operation of the local feature part.

[0066] Specifically, the present invention first determines the selected feature area based on the features diffused by the key feature central area diffusion module, but the selected part may ignore the important surrounding information in order to focus on the key information in the image. Therefore, in order to expand the receptive field of the model while paying attention to more key information, the present invention increases the feature area of ​​interest through the regional feature radiation mechanism to assist local features to improve the performance of the final network model. After the feature central area diffusion module determines the feature area, the present invention gathers these features together and determines the boundaries of these features. In order to expand the receptive field of this part of the features, the present invention expands the current area to the boundaries of each side of the irregular image to expand the scope of the entire feature selection area, and uses this part of the features as part of the local features for subsequent operations.

[0067] On this basis, in order to display the local features required for using context information to generate, the present invention uses a square ring partitioning strategy to divide the processed features into regions, thereby generating local features and global features for joint representation learning. The center of the divided image is approximately aligned with the center of the feature map. Therefore, the square ring partitioning strategy divides the entire image into blocks according to the distance from the center of the image. Although the image content under different viewing angles will change greatly, the various regions in the cross-view image are also approximately spatially aligned after division, which can also increase the feature similarity of each part after blocking, thereby improving the performance of the final model. Therefore, the present invention selects this part of the features as local features and global features to form the final model features together in order to effectively complete the cross-view geolocation task.

[0068] After determining these two parts of features, the original global features, the features generated by the key feature central area diffusion module, and the features generated by the edge radiation mechanism are added to form a new feature with the same size as the original global feature, and the newly generated features are passed through an MLP layer to further expand the receptive field of the features.

[0069] c. Loss function

[0070] In order to better train the proposed joint representation learning network based on feature center area diffusion and aggregation, the present invention adopts a set of hybrid loss functions, including two parts, namely the cross entropy loss function L cro and the adaptive triple InfoNCE loss function L tri_Info The overall loss function of training the network can be expressed as follows:

[0071] L total =L cro +L tri_Info (5)

[0072] The cross entropy loss function is widely used in classification tasks. Therefore, the cross entropy loss function is introduced to assist the model in training. The cross entropy loss function L cro The definition is as follows:

[0073]

[0074] in, is the logit score of the true geo-tag y, for The predicted probability of belonging to geographic label y.

[0075] In addition, in order to better assist the model in obtaining discriminative features, the present invention designs an adaptive triple INFONCE loss function to constrain the network model, so that the model can obtain better performance. The adaptive triple INFONCE loss function can be defined as follows:

[0076]

[0077] Where q is the encoded query feature and R is a set of reference encoded features. + is the positive sample of query feature q, r - is the negative sample of the query feature q. τ is a hyperparameter that can be learned or set to a static value. In addition, m f is margin,α f is the weight parameter.

[0078] To this end, the overall adaptive triple INFONCE loss function L of the training network model is tri_Info It can be expressed as follows:

[0079]

[0080] d. Image retrieval

[0081] Since the proposed network model is a dual-branch network structure, in the test phase, the reference image is first input into the corresponding branch network to obtain features and generate the final features through the classifier. Then, these features are stored as a reference library for query. On this basis, the features to be queried are input into the network of another branch to obtain the features of the input image. After that, the features of the input image are compared with the features in the reference query library, so as to find the images of the same scene at different perspectives that are closest to the query image in the query library to complete the geolocation task under cross-perspective.

[0082] In order to verify the effectiveness of the algorithm in terms of cross-view geolocation accuracy, the present invention is tested on the University-1652 dataset. According to the experimental results, the algorithm proposed in the present invention has Recall@1 and AP results of 92.79% and 93.91% respectively when retrieving satellite images from drone images, and Recall@1 and AP results of 95.58% and 92.17% respectively when retrieving drone images from satellite images.

Claims

1. A cross-view geolocation method based on feature center area diffusion and edge radiation, characterized in that: The steps include: Acquire images from different platform perspectives and input them into a backbone network; the backbone network is a dual-branch architecture, each branch has the same structure and the backbone network weights are shared; each branch structure includes a backbone network, a key feature central area diffusion module, an edge radiation mechanism and a classifier module; The backbone network is based on ConvNeXt-base; images from different platform perspectives are input into the backbone network to extract global features of the image; The global features are input into the classifier module and the key feature central area diffusion module respectively; The global feature, the feature obtained by the key feature central area diffusion module and the feature obtained by the edge radiation mechanism are added together to form a new feature with the same size as the original global feature. The new feature is input into the classifier module after passing through an MLP layer. The overall backbone network is optimized based on the loss function, and the optimized backbone network is used to solve the cross-view geolocation problem; During the testing phase, the reference image is first input into the corresponding branch network to obtain features and the final features are generated through the classifier, and these features are stored as a reference library for queries; on this basis, the features to be queried are input into another branch of the network to obtain the features of the input image; the features of the input image are compared with the features in the reference query library, so as to find the images of the same scene at different perspectives that are closest to the query image in the query library to complete the cross-perspective geolocation task.

2. The cross-viewing geolocation method based on characteristic central area diffusion and edge radiation according to claim 1 is characterized in that: The backbone network extracts global features as follows: Among them, x i is the input image, For the input image x i Extracted global features.

3. The cross-viewing geolocation method based on characteristic central area diffusion and edge radiation according to claim 2 is characterized in that: The key feature central area diffusion module is specifically: The key feature center area diffusion module extracts the key features from the global features, takes the key features as the center, and further diffuses the key features; The key feature center area diffusion module uses the feature heat map and feature similarity to determine which n key features are selected as the center area; Based on the relationship between the feature heat map and the feature similarity, the heat map judgment matrix and the similarity judgment matrix are obtained respectively; the heat map judgment matrix and the similarity judgment matrix are added and sorted from high to low to select the initial key feature center area; During the selection process, the features in the adjacent nine-square grid as the central area are deleted, and other regional features with high addition values ​​are used to supplement them, so as to determine the final key feature center; After determining the centers of these key features, they are diffused to the surrounding areas based on the central areas of these key features. During the diffusion process, all central areas diffuse to the surrounding areas in a nine-square grid manner at the same time. When the diffusion range of the feature overlaps with the diffusion range of other features or touches the boundary, the diffusion stops. Finally, the diffused features are combined together to form a component of the local information part.

4. The cross-viewing geolocation method based on characteristic central area diffusion and edge radiation according to claim 3 is characterized in that: The heat map judgment matrix is ​​obtained as follows: The feature vector obtained by the CBAM algorithm of the global feature is averaged based on the channel to represent the heat value of the feature; Among them, t i,j represents the heat value of the ith row and jth column in the heat map judgment matrix, P k It represents the characteristic heat value of the kth channel of the feature vector corresponding to the current part, m and n are the length and width of the generated matrix; H is the heat map judgment matrix; S is the number of channels of the feature vector.

5. The cross-viewing geolocation method based on characteristic central area diffusion and edge radiation according to claim 3 is characterized in that: The similarity judgment matrix acquisition process is as follows: the global features are averaged based on the channel and then divided into multiple small blocks to form local features, and the cosine similarity between the global features and each local feature is calculated to form a similarity judgment matrix A: Where D[·] represents the process of calculating feature similarity. Cosine similarity is used to calculate the similarity between global features and local features. m and n are the length and width of the generated matrix.

6. The cross-viewing geolocation method based on characteristic central area diffusion and edge radiation according to any one of claims 3 to 5, characterized in that: The edge radiation mechanism processes the features obtained by the key feature central area diffusion module, and the specific process is as follows: After the key feature center area diffusion module determines the feature center area, these key features are gathered together and the boundaries of these features are determined; since the gathered features are not of a relatively regular size, the boundary range of the features is determined based on the current irregular area and expanded to the boundaries of each side of the irregular image, thereby expanding the range of the entire feature selection area and generating edge radiation features; A square ring division strategy is used to divide the edge radiation feature into regions to generate local features; the square ring division strategy divides the entire edge radiation feature into blocks according to the distance from the center of the image.

7. The cross-viewing geolocation method based on characteristic central area diffusion and edge radiation according to claim 6, characterized in that: The loss function is divided into a cross entropy loss function L cro and the adaptive triple INFONCE loss function L tri_Info , the loss function is expressed as follows: L total =L cro +L tri_Info (6) The cross entropy loss function L cro The definition is as follows: in, is the logit score of the true geo-tag y, for The predicted probability of belonging to geographic label y; The adaptive triple INFONCE loss function is used to constrain the backbone network; the adaptive triple INFONCE loss function is defined as follows: Where q is the encoded query feature, R is a set of reference encoded features; r + is the positive sample of query feature q, r - is the negative sample of the query feature q; τ is a hyperparameter, which is a static value, m f is margin, α f is the weight parameter.

8. The cross-viewing geographic positioning method based on characteristic central area diffusion and edge radiation according to claim 7, characterized in that: The images from different platform perspectives include satellite images and drone perspective images, and the adaptive triple INFONCE loss function L tri_Info It is expressed as follows: is the adaptive triple INFONCE loss function for satellite images, Representing the adaptive triple INFONCE loss function for drone-view images.

9. The cross-viewing geographic positioning method based on characteristic central area diffusion and edge radiation according to claim 8, characterized in that: The classifier module consists of a fully connected layer, a batch normalization layer, a Dropout layer and a classification layer.

Images