A multi-view target geolocation method based on multi-layer local patterns

Abstract

The present invention discloses a multi-perspective target geolocation method based on multi-layer local patterns. First, theoretical modeling is performed; then a multi-layer local pattern cross-attention network is constructed, the network is based on a dual-branch architecture, and each branch consists of a backbone, a cross-attention block, a partitioned pooling block, and a multi-layer aggregation block; then the multi-layer local pattern cross-attention network is trained; multi-perspective target geolocation is achieved through the trained multi-layer local pattern cross-attention network. Existing methods usually focus on extracting fine-grained information from the final feature map, while ignoring the importance of the intermediate layer output. In the present invention, feature maps of different layers are first extracted, and cross-attention blocks are used to establish the correlation between feature map information of different layers, thereby improving the semantic expression of shallow features. The present invention proposes a multi-layer aggregation block to aggregate the highly correlated feature blocks obtained by segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of neural networks and aims to solve the matching problem of drone-view images and satellite-view images, and specifically relates to a multi-view target geographic positioning method based on multi-layer local patterns. Background Art

[0002] Cross-view geolocation is a task that requires finding images in a satellite image library that have the same location as a given drone-view image, and vice versa. Satellite-view images contain GPS data as geotags. Therefore, drones can effectively determine the location of the area to be retrieved. Currently, cross-view geolocation has attracted considerable attention from researchers in areas such as autonomous driving, drone navigation, target detection, and precise positioning. Deep learning-based cross-view geolocation can be considered a powerful tool for GPS to achieve more accurate geolocation. Furthermore, when the GPS system is unavailable, the cross-view geolocation model can be used independently to provide more robust results. The key to cross-view geolocation is to capture the most salient cues in the scene image to represent different target locations, that is, to capture salient landmarks. However, due to the inherent differences in visual appearance of images collected from different viewpoints (i.e., drones and satellites), it remains a challenging task.

[0003] Early bird's-eye view geolocation usually uses traditional image processing pipelines, gradient-based methods and hand-crafted features to obtain truly matching image pairs, but the above methods are time-consuming and labor-intensive, and the matching accuracy is not high. With the development of deep learning, cross-view geolocation has developed rapidly. A classic method is to apply a pre-trained convolutional neural network as the backbone to extract image descriptors. Then, a classifier layer is used to aggregate correctly matched drone and satellite image pairs in the feature space. Wang proposed a local pattern-based network that uses pre-trained CNN to extract features and fully mine contextual information. Zheng proposed a network model using dual-branch shared weights, which helps to learn perspective-invariant representations of image pairs.

[0004] Due to its powerful global modeling ability and ability to extract global contextual information, Transformer has attracted more and more attention in visual tasks. The backbone network based on Transformer has become the main choice. According to this scheme, some methods explicitly use contextual information to enhance feature expression. Zhu took advantage of the global information modeling and explicit location information encoding of the attention mechanism and proposed the first pure Transformer method for bird's-eye view geolocation. Dai proposed a Transformer framework that combines segmentation and feature alignment methods, further demonstrating the applicability of the attention mechanism in the field of cross-view geolocation. Therefore, the attention mechanism is also widely used to encourage the network to learn key clues from images. In DenseNet, the design that the output of each layer is connected to the output of all previous layers shows the importance of intermediate layer features. Recently, Lin used fine-grained key points in the intermediate feature map to promote model learning in the next stage.

[0005] However, previous related methods only consider mining fine-grained information from the final feature map that lacks low-level cues such as texture and edges, and do not consider extracting rich low-level cues in intermediate layer features. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention provides a multi-view target geographic positioning method based on multi-layer local patterns.

[0007] In this paper, we propose an end-to-end network, named Multi-layer Local Pattern Cross-Attention Network (MLPCAN). Our MLPCAN mines fine-grained information from feature maps at different layers to improve the accuracy of multi-view object geolocalization. Specifically, based on a two-branch architecture, MLPCAN mainly consists of (a) Cross-Attention Block (CAB), (b) Partition Pooling Block (PPB), and (c) Multi-Aggregation Block (MAB). Instead of directly utilizing feature maps extracted by the forward pass, our network employs Cross-Attention Blocks to highlight semantic representations in features from previous layers by using the final features as feedback indicators. By correlating information from previous and subsequent layers, lower-layer features can adaptively enhance discriminative regions while preserving lower-order cues such as color and texture. Similar to LPN, PPB uses a square ring partitioning strategy to divide feature maps at different layers into an equal number of contextual blocks. Due to inconsistent receptive fields, the information between corresponding blocks in different layers is inconsistent. Our MAB uses an asymmetric fusion strategy to coarsely align block features at different layers, effectively optimizing MLPCAN by leveraging fine-grained contextual information.

[0008] A multi-view target geolocation method based on multi-layer local patterns comprises the following steps:

[0009] Step 1: theoretical modeling;

[0010] Step 2: Construct a multi-layer local pattern cross attention network (MLPCAN);

[0011] The multi-layer local pattern crisscross attention network is based on a two-branch architecture, where each branch consists of a backbone, a crisscross attention block (CAB), a partitioned pooling block (PPB), and a multi-layer aggregation block (MAB).

[0012] Step 3: Train the multi-layer local pattern cross attention network;

[0013] During training, the classifier is used to predict the label of each local feature, and the network optimization is achieved by minimizing the sum of the cross entropy losses of all branches;

[0014] Step 4: Implement multi-view target geolocation through the trained multi-layer local pattern cross-attention network;

[0015] Furthermore, the specific method of step 1 is as follows;

[0016] Given a geolocation dataset, denote the input image as x and the corresponding label as y. Apply subscript z to denote the collection of data x. z platforms, and z∈{1,2}. Specifically, x1 represents samples from satellite view, and x2 represents drone view data. The labels y∈[1,C], where C represents the number of categories. For cross-view geolocalization, a mapping function is learned to project images from different platforms into a shared semantic space. Within this shared semantic space, images at the same location are close together, while images at different locations are separated.

[0017] Furthermore, the proposed MLPCAN consists of two branches, the drone view branch and the satellite view branch. Due to the similarity between the drone view and the satellite view, the two branches share weights and set the same feature extraction method. Specifically, the pre-trained SwinTransformerV2-Tiny (SwinV2-T) is used as the backbone of the two branches to extract features. SwinV2-T consists of a patch partition layer and four stages (S i ,i∈{1,2,3,4}), each stage consists of a merging layer and several stacked Swin Transformer Blocks (STBs). The feature scale of each stage is and Where H and W represent the height and width of the input image, respectively, and C represents the number of channels. Three different feature maps are obtained from the S3 stage as input to the cross attention block (CAB). The feature extraction process can be formulated as: n =F backbone (x) n=1,2,3, where f1 and f2 are shallow features and f3 is a deep feature. In order to improve the semantic expression of shallow features f1 and f2, f1 and f2 are used as keys and f3 as queries to perform cross attention block (CAB). The cross attention block (CAB) includes a multi-head attention block and a feedforward network. The multi-head attention block first uses three linear layers to map two input features (f1 and f3, or f2 and f3) to obtain query query (Q), key key (K) and value value (V), and obtains the attention weight Attention(Q,K,V)=softmax(QK T )V, which is then fused with f1 through Linear and layer normalization layers.

[0018] The feedforward network includes linear layers, activation layers, and layer normalization to enhance stability. Feature maps f1, f2, and f3 are extracted from layer1, layer2, and layer6 of S3. The overall processing of the cross-attention block is expressed as:

[0019]

[0020]

[0021] in and f' i are the outputs of the multi-head attention block and the feedforward network, respectively. Concat(·) indicates that the features are concatenated along the channel dimension.

[0022] The deep feature f3 and the shallow feature after the cross attention block are processed by the partition pooling block (PPB) respectively. The partition pooling block (PPB) is used to explicitly mine the context information to extract the salient target. In the partition part, the input global features f3 and f' are partitioned by the ring partition process. i (16×16×384) is divided into four non-overlapping features, named

[0023] The superscript j represents the jth partition, and its value corresponds to the distance between the partition and the center of the input image. The smaller the superscript, the closer to the center. Then, an average pooling layer is applied to each partition. Some features compressed to 384 dimensions The process is publicly expressed as:

[0024]

[0025]

[0026] where f n Represents input features, Split represents square ring partition, P avg Represents an average pooling operation.

[0027] Partial features are obtained from different layers. In order to obtain a more discriminative final descriptor, the partial features are fused. n The receptive field is different, and the one obtained after PPB The distribution of is inconsistent, so an asymmetric fusion strategy is proposed to use features with large receptive fields to fuse features with small receptive fields to achieve feature alignment. This strategy is implemented by the proposed multi-layer aggregation block (MAB), and the formula is as follows:

[0028]

[0029] Where <·> represents the channel fusion operation. Then, the final feature vector k is obtained j , where k 1 , k 2 , k 3 The dimensions of k are 1152. 4 The dimension is 1920.

[0030] After processing by multi-layer aggregation block (MAB), the feature vector k is obtained j However, MLPCAN has two branches, a total of eight sets of local features, and their distribution is different, so they cannot be used directly for matching. A classifier is introduced to map the eight sets of local features to a shared feature space. The classifier consists of a linear layer, a batch normalization layer, a dropout layer, and a classification layer. In the case of , the classifier generates a column vector The dimension is equal to the number of geographic tag categories C.

[0031] Furthermore, the specific method of step 3 is as follows;

[0032] During training, the classifier is used to predict the label of each local feature, and network optimization is achieved by minimizing the sum of the cross entropy losses of all branches, as follows:

[0033]

[0034]

[0035] in is the logarithm of the drone geotag y score. In equation (6), the normalized probability score is obtained Right now The predicted probability of . Equation (7) represents the cumulative loss of different parts and two branches. During the test phase, the image representation before the classification layer is obtained in the classifier. These features are then concatenated as the final discriminant feature with a dimension of 2048. The similarity between the query and the images in the candidate gallery is then calculated using the cosine distance. The final retrieval results are sorted based on the similarity.

[0036] The beneficial effects of the present invention are as follows:

[0037] Existing methods usually focus on extracting fine-grained information from the final feature map, while ignoring the importance of intermediate layer outputs. In this paper, we first extract feature maps from different layers and use cross-attention blocks (CABs) to establish correlations between feature map information from different layers, improving the semantic expression of shallow features. The present invention proposes a multi-layer aggregation block (MAB) to aggregate highly correlated feature blocks obtained by segmentation. Tests on public datasets show that it significantly improves the accuracy of geolocation and achieves competitive results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the framework of the multi-layer local pattern cross attention network (MLPCAN) of the present invention;

[0039] Figure 2 A heat map comparison of whether CAB is used in the embodiment of the present invention;

[0040] Figure 3 Schematic diagram of MAB and DAB according to an embodiment of the present invention;

[0041] Figure 4 The figure shows a heat map comparison of whether or not MAB is used in the embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described below with reference to the accompanying drawings and embodiments.

[0043] A multi-view target geolocation method based on multi-layer local patterns comprises the following steps:

[0044] Step 1: theoretical modeling;

[0045] Given a geolocation dataset, denote the input image as x and the corresponding label as y. Apply subscript z to denote the collection of data x. zplatforms, and z∈{1,2}. In particular, x1 represents samples from satellite views and x2 represents drone view data. The labels y∈[1,C], where C represents the number of categories. For example, the dataset includes 701 buildings, and each building contains multiple images. We divide the 701 buildings into 701 different indices, each index represents a category, that is, labels y∈[1,701]. For cross-view geolocalization, a mapping function is learned to project images from different platforms into a shared semantic space. In this shared semantic space, images at the same location are close, while images at different locations are separated from each other.

[0046] Step 2: Construct a multi-layer local pattern cross attention network (MLPCAN);

[0047] The proposed MLPCAN consists of two branches, the UAV view branch and the satellite view branch. Due to the similarity of the patterns between the UAV view and the satellite view, the two branches share weights and set the same feature extraction method, such as Figure 1 Specifically, we use the pre-trained SwinTransformerV2-Tiny (SwinV2-T) as the two-branch backbone to extract features. SwinV2-T consists of a patch partition layer and four stages (S i ,i∈{1,2,3,4}), each stage consists of a merging layer and several stacked Swin Transformer Blocks (STBs). The feature scale of each stage is and Where H and W represent the height and width of the input image respectively, and C represents the number of channels. Figure 1 As shown, we obtain three different feature maps from the S3 stage as the input of the cross attention block (CAB). The feature extraction process can be formulated as: n =F backbone (x) n=1,2,3, where f1 and f2 are shallow features and f3 is a deep feature. In order to improve the semantic expression of shallow features f1 and f2, we use f1 and f2 as keys and f3 as query to perform cross attention block (CAB). Figure 1 The proposed Cross Attention Block (CAB) is presented, which includes a multi-head attention block and a feedforward network. The multi-head attention block first uses three linear layers to map two input features (f1 and f3, or f2 and f3) to obtain the query query (Q), key key (K) and value value (V), and obtains the attention weight Attention(Q,K,V) = softmax(QK T)V, which is then fused with f1 through Linear and layer normalization layers.

[0048] Intuitively, the attention operation selects relevant information by measuring the similarity between the query element and each key element. The output vector is the sum of the similarity score and the value vector. Consequently, if the similarity is high, the relevant information is extracted from the value vector. This process is also known as "message passing" in graph neural networks.

[0049] The feedforward network consists of linear layers, activation layers, and layer normalization to enhance stability. In this paper, we extract feature maps f1, f2, and f3 from layer1, layer2, and layer6 of S3. The overall processing of the cross-attention block is expressed as:

[0050]

[0051]

[0052] in and f' i are the outputs of the multi-head attention block and the feedforward network, respectively. Concat(·) indicates that the features are connected along the channel dimension. In order to accurately reflect the focus area of ​​the model, we provide heat maps generated by different layers with or without CAB on UAV and satellite platforms for comparison. Figure 2 As shown in (a), CAB activates more prominent target regions. Furthermore, the focal region is broader and more accurate than shallow features. Deep features guide shallow features and improve their semantic representation. Furthermore, we find that improving semantic representation does not result in a loss of original information. Therefore, the working principle of CAB is consistent with our previous discussion.

[0053] The deep feature f3 and the shallow feature after the cross attention block are processed by the partition pooling block (PPB) respectively. The partition pooling block (PPB) is used to explicitly mine the context information to extract the salient target. In the partition part, we use the square ring partition process to partition the input global features f3 and f' i (16×16×384) is divided into four non-overlapping features, named

[0054] The superscript j represents the jth partition, and its value corresponds to the distance between the partition and the center of the input image. The smaller the superscript, the closer to the center. Then, we apply an average pooling layer to each partition Some features compressed to 384 dimensions The process is publicly expressed as:

[0055]

[0056]

[0057] where f n Represents input features, Split represents square ring partition, P avg Represents an average pooling operation.

[0058] In this way, we obtain partial features from different layers. In order to obtain a more discriminative final descriptor, it is necessary to fuse the partial features. n The receptive field is different, and the one obtained after PPB The distribution of has inconsistencies, for example, and Although they all belong to the same part, their feature representations may be different. Direct fusion methods may cause distortion of texture information. To address this limitation, we propose an asymmetric fusion strategy that utilizes features with large receptive fields, such as To fuse the features with small receptive field, and To achieve the effect of feature alignment. Figure 3 (a). This strategy is achieved by the proposed multi-layer aggregation block (MAB), as follows:

[0059]

[0060] Where <·> represents the channel fusion operation. Then, we obtain the final feature vector k j , where k 1 , k 2 , k 3 The dimensions of k are 1152. 4 The dimension of is 1920. We adopt Direct Aggregation Block (DAB) to represent the above direct fusion method and briefly illustrate the effectiveness of the proposed MAB. Figure 3 Shown, demonstrating the difference between MAB (a) and DAB (b). Figure 4 The heatmaps generated using MAB (c) and DAB (b) are shown, with (a) as the input. It can be observed that the application of MAB activates more accurate geographical targets than DAB.

[0061] After processing by multi-layer aggregation block (MAB), we obtain the feature vector k j However, MLPCAN has two branches, a total of eight groups of local features, and their distributions are different and cannot be used directly for matching. To solve this problem, we introduce a classifier that maps the eight groups of local features into a shared feature space, which allows features with similar semantic labels to have closer distances. Figure 1As shown, the classifier consists of a linear layer, a batch normalization layer, a dropout layer, and a classification layer. In the case of , the classifier generates a column vector The dimension is equal to the number of geographic tag categories C.

[0062] Step 3: Train the multi-layer local pattern cross attention network;

[0063] During training, the classifier is used to predict the label of each local feature, and network optimization is achieved by minimizing the sum of the cross entropy losses of all branches, as follows:

[0064]

[0065]

[0066] in is the logarithm of the drone geotag y score. In equation (6), the normalized probability score is obtained Right now The predicted probability of . Equation (7) represents the cumulative loss of different parts and dual branches. In the test phase, we obtain the image representation before the classification layer in the classifier, such as Figure 1 We then concatenate these features to form the final discriminant feature, whose dimension is 2048. We then use the cosine distance to calculate the similarity between the query and the images in the candidate gallery. The final retrieval results are ranked based on similarity.

[0067] Step 4: Network test and evaluation;

[0068] For evaluation purposes, we conduct extensive comparisons with benchmark datasets, including University-1652 and SUES-200. Specific experimental data are shown in Tables 1 and 2.

[0069]

[0070] Table 1

[0071]

[0072] Table 2

[0073] In bird’s-eye view geolocation, commonly used evaluation metrics include Recall@k (R@K) and Average Recall (AP). R@K represents the proportion of correctly matched images in the top K of the ranking list. A higher score indicates better retrieval performance of the network. The relevant formula is expressed as Where N represents the number of true matches in the top K search results. The average recall (AP) reflects the precision and recall of the retrieval performance.

[0074] To validate the effectiveness of the proposed MLPCAN, we conducted extensive experiments on two benchmark datasets, University-1652 and SUES-200. Our method achieved an accuracy of 91.21% (recall@1) in the drone-to-satellite task, outperforming the baseline by 32.72% and 29.16% in the drone-to-satellite and satellite-to-drone tasks on the University-1652 dataset, respectively. On the SUES-200 dataset, it achieved an accuracy of 91.94% (recall@1) and 93.63% (AP), respectively, in the drone-to-satellite task, representing increases of 24.28% and 21.59% compared to the baseline.

[0075] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0076] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A multi-view target geolocation method based on multi-layer local patterns, characterized in that: The steps include: Step 1: theoretical modeling; Step 2: Construct a multi-layer local pattern cross attention network MLPCAN; The multi-layer local pattern cross attention network is based on a two-branch architecture, where each branch consists of a backbone, a cross attention block CAB, a partitioned pooling block PPB, and a multi-layer aggregation block MAB. Step 3: Train the multi-layer local pattern cross attention network; During training, the classifier is used to predict the label of each local feature, and the network optimization is achieved by minimizing the sum of the cross entropy losses of all branches; Step 4: Implement multi-view target geolocation through the trained multi-layer local pattern cross-attention network; The proposed MLPCAN consists of two branches, the UAV view branch and the satellite view branch. Due to the similarity between the UAV and satellite views, the two branches share weights and set the same feature extraction method. Specifically, the pre-trained SwinTransformerV2-Tiny is used as the two-branch backbone to extract features; SwinV2-T contains a patch partition layer and four stages , each stage consists of a merging layer and several stacked Swin Transformer Blocks; the feature scale of each stage is C, 2C, C and C, where H and W represent the height and width of the input image respectively, and C represents the number of channels; three different feature maps are obtained from the S3 stage as the input of the cross attention block CAB. The feature extraction process can be formulated as follows: ,in and It is a shallow feature. For deep features; in order to improve shallow features and The semantic expression of and As a key, As a query, the cross attention block CAB is executed; the cross attention block CAB includes a multi-head attention block and a feedforward network; the multi-head attention block first uses three linear layers to map the two input features and ,or and To get the query query(Q), key key(K) and value value(V), and get the attention weight , after the Linear and layer normalization layers and to integrate; The feedforward network includes linear layers, activation layers, and layer normalization to enhance stability; feature maps are extracted from layer1, layer2, and layer6 of S3. ; Cross attention block The body processing process is expressed as: in and are the outputs of the multi-head attention block and the feedforward network, Represents features connected along the channel dimension; Deep features The shallow features after the cross attention block are processed by the partition pooling block PPB respectively; The context information is explicitly mined through the partition pooling block PPB to extract salient targets; in the partition part, the input global features are partitioned into square rings. Divided into four non-overlapping features, named ; The superscript j represents the jth partition, and its value corresponds to the distance between the partition and the center of the input image; the smaller the superscript, the closer to the center; then, the average pooling layer is applied to each partition Some features compressed to 384 dimensions ; The process is publicly expressed as: in represents the input features, represents a square ring partition, represents the average pooling operation; Obtain partial features from different layers; fuse these features to obtain a more discriminative final descriptor; Due to the characteristics The receptive field is different, and the one obtained after PPB The distribution of is inconsistent. Using features with large receptive fields, Fusion of features with small receptive fields, To achieve the effect of feature alignment; this strategy is implemented by the proposed multi-layer aggregation block MAB, the formula is as follows: （5） Where <∙> represents the channel fusion operation; then, the final feature vector is obtained ,in , , The dimensions are all 1152, The dimension is 1920; After processing by multi-layer aggregation block MAB, the feature vector is obtained ;However, MLPCAN has two branches, a total of eight groups of local features, and their distribution is different, so they cannot be used directly for matching; a classifier is introduced to map the eight groups of local features to a shared feature space. The classifier consists of a linear layer, a batch normalization layer, a dropout layer, and a classification layer; when the local features are input In the case of , the classifier generates a column vector ; The dimension is equal to the number of geographic tag categories C.

2. The multi-view target geographic positioning method based on multi-layer local patterns according to claim 1, characterized in that: Step 1: The specific method is as follows; Given a geolocation dataset, denote the input image as x, y as the corresponding label; apply the subscript z to denote the collection data platform, and z∈{1,2}; in particular, represents a sample from a satellite view, represents drone view data; the label y∈[1, C], where C represents the number of categories; for cross-view geolocalization, a mapping function is learned to project images from different platforms into a shared semantic space; within this shared semantic space, images at the same location are close, while images at different locations are separated from each other.

3. The multi-view target geographic positioning method based on multi-layer local patterns according to claim 2, characterized in that: Step 3: The specific method is as follows; During training, the classifier is used to predict the label of each local feature, and network optimization is achieved by minimizing the sum of the cross entropy losses of all branches, as follows: （6） （7） in is the logarithm of the drone geotag y score; in equation (6), the normalized probability score is obtained ,Right now The predicted probability of Equation (7) represents the cumulative losses of different parts and dual branches; During the testing phase, the image representation before the classification layer is obtained in the classifier; Then, these features are concatenated as the final discriminant feature with a dimension of 2048. The cosine distance is then used to calculate the similarity between the query and the images in the candidate gallery; the final retrieval results are sorted based on the similarity.

Images