Progressive cross-view-angle image geographic positioning method based on spatial feature aggregation and position perception

Abstract

The invention relates to the technical field of image geographic positioning, in particular to a progressive cross-view-angle image geographic positioning method based on spatial feature aggregation and position awareness, which comprises the following steps: acquiring paired satellite images and ground panoramic images as sample images, and constructing a training sample set; a cross-view image geographic positioning model composed of a double-branch backbone network and a fine-grained prediction network is constructed, two branches of the double-branch backbone network are a satellite processing branch and a ground panorama processing branch, and the fine-grained prediction network is composed of a bird's-eye view image transformation module and a position sensing prediction module; inputting training samples in the training sample set into the cross-view image geographic positioning model, and performing iterative training on the cross-view image geographic positioning model until convergence; and inputting a to-be-positioned paired satellite image and ground panoramic image into the trained cross-view-angle image geographic positioning model to obtain a positioning result output by the trained cross-view-angle image geographic positioning model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image geolocation, and in particular to a method for spatial feature aggregation and location-aware progressive cross-view image geolocation. BACKGROUND

[0002] Image geolocation is a positioning technology based on computer vision, whose main purpose is to match the query image with unknown geographical information with the reference image library with geographical position marks, so as to determine the actual geographical position of the query image. This technology can provide an auxiliary positioning support for traditional signal positioning methods in special environments such as GPS denial, and plays an important role in scenes such as unmanned aerial vehicle navigation, intelligent driving, unmanned delivery, and target tracking. With the advantages of easy acquisition of ground panoramic maps and wide coverage of satellite maps, cross-view image geolocation technology based on ground panoramic maps and satellite maps has attracted more and more research teams.

[0003] When performing cross-view image geolocation based on ground panoramic maps and satellite maps, a ground panoramic map taken from a ground view (such as an unmanned vehicle) is usually selected as a query image, and the query image is matched with a pre-constructed satellite reference image library with geographical information marks, so as to determine the specific position represented by the ground panoramic map. The huge appearance difference between the ground panoramic map and the satellite image pair and the possible central position offset bring great challenges to this task. The image geolocation problem is usually regarded as a retrieval task, and with the rapid development of deep learning technology, the twin network architecture with CNNs as feature extractors and the fusion of spatial attention has gradually become the mainstream network backbone for cross-view image geolocation methods. Hu et al. successfully integrated the NetVLAD layer into the output of the VGG network, proposed CVM-Net for cross-view geolocation tasks, and introduced spatial attention to improve the representation ability of image features; Yang et al. fused the Transformer module with CNN, proposed the L2LTR network, which can better learn the global context and location-aware representation of images; Zhu et al. also proposed a pure Transformer-based double-branch cross-view image geolocation network TransGeo, designed a two-stage training strategy, so that the model pays more attention to image blocks with significant features, thereby generating more discriminative image descriptors.

[0004] There are also research teams focusing on reducing the difference between the two perspectives first, so as to achieve higher precision cross-perspective image matching. For example, Shi et al. proposed a polar coordinate transformation method, which converts satellite perspective images into ground perspective images by using the pixel correspondence between the overhead view and the ground panoramic image, and designed an SPE (Spatial Perception Location Embedding) module that can give different weight factors to different target semantics in space; Li et al. proposed an inverse polar coordinate transformation method to convert ground images into satellite perspective images.

[0005] The above methods are all based on the alignment of ground images and satellite images. In the case of offset or misalignment, the effectiveness will be reduced. Since it is difficult to achieve direct positioning in the misaligned state of the image, this task is gradually divided into two sub-directions: coarse-grained retrieval positioning in the misaligned state and fine-grained position prediction in the matching relationship. Shi et al. use Transformer to obtain the BEV feature map of the ground image, and then transform the BEV feature map according to all possible ground camera poses, and compare it with the corresponding satellite image to determine the offset. Shi et al. also proposed a method to solve the ground position pose by iterative optimization, which transforms the satellite image into the ground image through the homography matrix, and then extracts the features of the transformed satellite image and the ground image, and further estimates the offset result.

[0006] However, although various research teams have made many improvements, the cross-perspective image geolocation technology of the ground panoramic image and the satellite image currently proposed still has the following limitations.

[0007] First, the current cross-perspective image geolocation method mainly realizes ground panoramic image geolocation for the center-aligned cross-perspective image pair, and uses image perspective transformation method, global feature extractor, and spatial attention to realize feature representation. The positioning effect is poor for image pairs with center position offset.

[0008] Second, the current cross-perspective image geolocation method separates coarse-grained image retrieval and fine-grained offset prediction, and lacks a fusion method that can simultaneously realize the two tasks, which greatly limits the improvement of positioning effect. SUMMARY

[0009] Therefore, the embodiments of the present application propose a spatial feature aggregation and location-aware progressive cross-perspective image geolocation method, which can effectively fuse fine-grained position offset prediction while performing coarse-grained image retrieval, and optimize the retrieval recall rate based on the results of offset prediction, realizing the end-to-end aggregation of two-stage tasks, and effectively improving the precision and robustness of cross-perspective image geolocation.

[0010] To achieve the above object, the embodiment of the present application proposes a spatial feature aggregation and location-aware progressive cross-view image geolocation method, which comprises: acquiring a pair of satellite images and ground panoramic images as sample images to construct a training sample set; constructing a cross-view image geolocation model composed of a double-branch backbone network and a fine-grained prediction network, two branches of the double-branch backbone network are a satellite processing branch and a ground panoramic processing branch, respectively used for global feature extraction of satellite images and ground panoramic images to obtain satellite global features and ground panoramic global features, the fine-grained prediction network is composed of a bird's eye view image transformation module and a location-aware prediction module, the bird's eye view image transformation module is used for bird's eye view angle transformation of the ground panoramic image based on coordinate system transformation to obtain a bird's eye view image, and the location-aware prediction module is used for location-aware prediction based on the satellite image and the bird's eye view image to obtain a location-aware prediction result; inputting the training sample in the training sample set into the cross-view image geolocation model, constructing an overall loss function based on the satellite global features, the ground panoramic global features and the location-aware prediction result, and iteratively training the cross-view image geolocation model by using the overall loss function, and obtaining the trained cross-view image geolocation model after meeting the training convergence condition; inputting a pair of satellite images and ground panoramic images to be positioned into the trained cross-view image geolocation model to obtain the positioning result output by the trained cross-view image geolocation model.

[0011] In order to achieve the above-mentioned purpose, the embodiment of the present application also proposes a progressive cross-view image geolocation system based on spatial feature aggregation and location awareness, the system including: a training sample set construction module, used to obtain paired satellite images and ground panoramic images as training sample images to construct a training sample set; a model construction module, used to construct a cross-view image geolocation model composed of a dual-branch backbone network and a fine-grained prediction network, the two branches of the dual-branch backbone network are satellite processing branch and ground panoramic processing branch, respectively used to extract global features of satellite images and ground panoramic images to obtain satellite global features and ground panoramic global features, the fine-grained prediction network consists of a bird's-eye view image transformation module and a location awareness prediction module, the bird's-eye view image transformation module is used to transform the ground based on the coordinate system. The cross-view image geolocation model is iteratively trained using the overall loss function. After the training convergence conditions are met, the trained cross-view image geolocation model is obtained. The model usage module is used to input the paired satellite images and ground panoramic images to be located into the trained cross-view image geolocation model to obtain the positioning results output by the trained cross-view image geolocation model.

[0012] In order to achieve the above-mentioned purpose, an embodiment of the present application also proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a progressive cross-view image geolocation method based on spatial feature aggregation and location awareness as described above.

[0013] In order to achieve the above-mentioned purpose, an embodiment of the present application also proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a progressive cross-view image geolocation method based on spatial feature aggregation and location awareness as described above.

[0014] The application provides a space feature aggregation and position-aware progressive cross-view image geolocation method, designs, constructs, trains and uses a cross-view image geolocation model to realize a cross-view image geolocation task. For a center misaligned satellite image and a ground panoramic image pair, coarse-grained retrieval matching and fine-grained offset prediction task fusion are realized, and cross-view image geolocation is performed based on a progressive metric learning idea, so that the positioning accuracy is effectively improved. The application applies a pseudo-twin stem network and a fine-grained prediction fusion strategy. The double-branch pseudo-twin network stem is responsible for coarse-grained reference image candidate set screening, the fine-grained prediction network realizes matching degree prediction and center offset estimation by using bird's-eye view image transformation and position-aware prediction, the two processes are fused at the same time, the geolocation capability of the cross-view image geolocation model for center misaligned image pairs is improved, and therefore the accuracy and robustness of the cross-view image geolocation are effectively improved.

[0015] Optionally, the satellite processing branch specifically consists of a first feature extraction module and a first space feature aggregation module, and the ground panoramic processing branch specifically consists of a second feature extraction module and a second space feature aggregation module.

[0016] The first feature extraction module and the second feature extraction module have the same structure and are both composed of the first four groups of convolution modules of ConvNeXt-T. The first four groups of convolution modules of ConvNeXt-T include an initial convolution layer and three consecutive convolution groups. The initial convolution layer has a convolution kernel size of 4x4, a step of 4 and an output channel number of 96. The three convolution groups are all composed of ConvNeXt-Blocks, but the number of Blocks in each convolution group is different. The first convolution group includes three ConvNeXt-Blocks, each Block includes a depth separable convolution with a size of 7x7, a dimension increasing convolution layer with a size of 1x1, a normalization layer, a GELU activation layer and a dimension decreasing convolution layer with a size of 1x1, and the output channel number is 96. The second convolution group also includes three ConvNeXt-Blocks, and the output channel number is 192. The third convolution group includes nine ConvNeXt-Blocks, and the output channel number is 384.

[0017] Let the satellite image be I s , the ground panoramic image be I g , I s and I g be input into the first feature extraction module and the second feature extraction module, respectively. The first feature extraction module is used for feature extraction on I s to obtain a satellite intermediate feature F s with a size of 1 / 16 of I s , and the second feature extraction module is used for feature extraction on I gFeature extraction is performed to obtain satellite intermediate features F g of 1 / 16 size I g ;

[0018] wherein, the size of F s and F g is CxHxW, C is the number of channels, H is the height, and W is the width.

[0019] Optionally, the first spatial feature aggregation module and the second spatial feature aggregation module have the same structure and are each composed of A independent local spatial attention units, each of which is used for significant feature extraction on the corresponding intermediate feature while keeping the number of front and rear feature channels unchanged.

[0020] The first spatial feature aggregation module performs maximum pooling along the channel dimension on F s to obtain a first significant feature mask M After two fully connected layers for feature mixing and selection, a local spatial attention mask M s2 is obtained. Finally, M s2 and F s are multiplied element by element, and each local spatial attention unit of the first spatial feature aggregation module outputs a feature vector f i s , i∈[0,A-1].

[0021] The outputs of each local spatial attention unit of the first spatial feature aggregation module are spliced in the channel dimension to obtain a satellite global feature f s of length AxC.

[0022] The second spatial feature aggregation module performs maximum pooling along the channel dimension on F g to obtain a second significant feature mask M After two fully connected layers for feature mixing and selection, a local spatial attention mask M g2 is obtained. Finally, M g2 and F g are multiplied element by element, and each local spatial attention unit of the second spatial feature aggregation module outputs a feature vector f i g , i∈[0,A-1].

[0023] The outputs of each local spatial attention unit of the second spatial feature aggregation module are spliced in the channel dimension to obtain a satellite global feature f g of length AxC.

[0024] Optionally, the bird's-eye view image transformation module is used to perform a bird's-eye view transformation on the ground panoramic image based on the transformation of the coordinate system to obtain a bird's-eye view image, including:

[0025] According to the imaging principle of the ground panorama, using the spherical coordinate system O sp The conversion relationship between the spatial rectangular coordinate system O1X1Y1Z1, the normalized equirectangular coordinate system X2O2Y2 and the ground pixel coordinate system o p The conversion relationship between them leads to the spherical coordinate system O sp The conversion relationship between the normalized rectangular coordinate system X2O2Y2;

[0026] Take the position of the upper left corner of the ground panorama as the ground pixel coordinate system o p The origin of the bird's-eye view pixel coordinate system is taken as the position of the upper left corner of the blank bird's-eye view image. b The origin of the spherical coordinate system O sp A section is made at the south pole of the image to serve as the bird's-eye view pixel coordinate system o b The projection plane, according to the bird's-eye view pixel coordinate system o b With the spherical coordinate system O sp The conversion relationship between the spherical coordinate system O sp The conversion relationship between the spatial rectangular coordinate system O1X1Y1Z1 and the spherical coordinate system O sp The conversion relationship between the normalized rectangular coordinate system O2O2Y2 and the spherical coordinate system O sp The next point Corresponding to the point in the normalized rectangular coordinate system X2O2Y2 With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them;

[0027] Points based on the normalized rectangular coordinate system X2O2Y2 With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them is used to derive the ground pixel coordinate system o p The next point With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them;

[0028] Based on the ground pixel coordinate system p The next point With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between the satellite image and the bird's eye view image is used to transform the bird's eye view angle of the ground panoramic image to obtain a bird's eye view image.

[0029] Optionally, the location-aware prediction module specifically consists of a feature extractor and a location-aware prediction unit, and the location-aware prediction module is configured to perform location-aware prediction based on the satellite image and the bird's eye view image to obtain a location-aware prediction result, including:

[0030] The feature extractor is configured to extract features from the satellite image and the bird's eye view image respectively to obtain satellite prediction features F ps and bird's eye view prediction features F pb .

[0031] The location-aware prediction unit is configured to perform location-aware prediction based on F ps and F pb , and then pass the matching probability Prob pd of the image pair and the possible center offset delta pd of the image pair through a multi-layer perception to obtain the location-aware prediction result.

[0032] Optionally, a total loss function is constructed based on the satellite global features, the ground panoramic global features and the location-aware prediction result, including:

[0033] f s and f g are mapped to the same feature space, the similarity between f s and f g is calculated by distance measurement, and a weighted soft-margin triplet loss function is constructed based on the calculated similarity to thereby shorten the distance between the matched image pairs and lengthen the distance between the unmatched image pairs.

[0034] The weighted soft-margin triplet loss function is represented by the following formula:

[0035] Loss coarse = ln{1+exp[α(d p -d n )]};

[0036] wherein, Loss coarse represents the weighted soft-margin triplet loss function, d p and d n respectively represent the Euclidean distances between the anchor point and the positive sample and the negative sample, and α is a preset hyperparameter.

[0037] A prediction joint loss function is constructed based on Prob pd and delta pd .

[0038] The prediction joint loss function is expressed by a formula as follows:

[0039]

[0040] wherein, Loss predit represents the prediction joint loss function, B is a batch size, ε is a very small positive number, delta gt is a center offset true value;

[0041] The overall loss function Loss coarse is constructed based on Loss predit and Loss total , and Loss total is expressed as:

[0042] Loss total = δ1·Loss coarse + δ2·Loss predit ;

[0043] wherein, δ1 and δ2 are weight coefficients of Loss coarse and Loss predit respectively, and δ1 + δ2 = 1.

[0044] Optionally, after obtaining the trained cross-view image geolocation model, the method further comprises:

[0045] inputting a test image into the trained cross-view image geolocation model, performing coarse-grained retrieval using a double-branch backbone network to obtain a candidate result set of TopN number, then performing prediction of matching probability and offset through a fine-grained prediction network to obtain the closest first K satellite image query results, and evaluating the coarse-grained image retrieval result using a recall rate index and evaluating the fine-grained offset prediction result using a mean error and a median error. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Figure 1 is a flowchart of a spatial feature aggregation and location-aware progressive cross-view image geolocation method provided in an embodiment of the present application;

[0048] Figure 2 is a structural diagram of a cross-view image geolocation model provided in an embodiment of the present application;

[0049] Figure 3 is a coordinate transformation relationship diagram of the bird's eye view image provided in an embodiment of the present application;

[0050] Figure 4 is a structure diagram of the location-aware prediction module provided in an embodiment of the present application;

[0051] Figure 5 is a comparison diagram of test results of the present method and other methods on the VIGOR dataset provided in an embodiment of the present application;

[0052] Figure 6 is a structure diagram of a spatial feature aggregation and location-aware progressive cross-view image geolocation system provided in another embodiment of the present application;

[0053] Figure 7 is a structure diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. However, those skilled in the art can understand that, in the embodiments of the present application, many technical details are proposed in order to make the readers better understand the present application. However, the technical solutions claimed by the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present application, and the embodiments can be combined and referenced with each other without contradiction.

[0055] An embodiment of the present application proposes a spatial feature aggregation and location-aware progressive cross-view image geolocation, which is applied to an electronic device. The electronic device can be a terminal or a server. The embodiments and the following embodiments are described by taking the server as an example. The implementation details of the spatial feature aggregation and location-aware progressive cross-view image geolocation proposed in the embodiment will be described in detail below. The following content only provides implementation details for easy understanding, and is not necessary for implementing the present solution.

[0056] The specific process of the spatial feature aggregation and location-aware progressive cross-view image geolocation proposed in the embodiment can be as shown in Figure 1 , which includes:

[0057] Step 11, acquiring a pair of satellite images and ground panoramic images as sample images to construct a training sample set.

[0058] In a specific implementation, the server first needs to obtain pairs of satellite images and ground panoramic images (i.e., satellite images and ground panoramic images of the same target) as sample images to form a training sample set.

[0059] In one example, the pairs of satellite images and ground panoramic images can be obtained from an open source website or by real shooting without considering the cost.

[0060] Step 12, a cross-view image geolocation model composed of a double-branch backbone network and a fine-grained prediction network is constructed, the two branches of the double-branch backbone network are a satellite processing branch and a ground panoramic processing branch, which are respectively used for global feature extraction of satellite images and ground panoramic images to obtain satellite global features and ground panoramic global features, the fine-grained prediction network is composed of a bird's eye view image transformation module and a location perception prediction module, the bird's eye view image transformation module is used for bird's eye view transformation of the ground panoramic image based on coordinate system transformation to obtain a bird's eye view image, and the location perception prediction module is used for location perception prediction based on the satellite image and the bird's eye view image to obtain a location perception prediction result.

[0061] In a specific implementation, after the server constructs the training sample set, a cross-view image geolocation model can be constructed. The cross-view image geolocation model is composed of a double-branch backbone network and a fine-grained prediction network, the two branches of the double-branch backbone network are a satellite processing branch and a ground panoramic processing branch, which are respectively used for global feature extraction of satellite images and ground panoramic images to obtain satellite global features and ground panoramic global features, the fine-grained prediction network is composed of a bird's eye view image transformation module and a location perception prediction module, the bird's eye view image transformation module is used for bird's eye view transformation of the ground panoramic image based on coordinate system transformation to obtain a bird's eye view image, and the location perception prediction module is used for location perception prediction based on the satellite image and the bird's eye view image to obtain a location perception prediction result.

[0062] In one example, the specific structure of the cross-view image geolocation model can be as shown in Figure 2 .

[0063] In one example, as shown in Figure 2 , the satellite processing branch is specifically composed of a first feature extraction module and a first spatial feature aggregation module, and the ground panoramic processing branch is specifically composed of a second feature extraction module and a second spatial feature aggregation module.

[0064] The structure of the first feature extraction module and the second feature extraction module is the same, both of which are composed of the first four convolution modules of ConvNeXt-T. The first four convolution modules of ConvNeXt-T include an initial convolution layer and three consecutive convolution groups. The convolution kernel size of the initial convolution layer is 4×4, the stride is 4, and the number of output channels is 96. The three convolution groups are all composed of ConvNeXt-Block, but the number of blocks in each convolution group is different. The first convolution group contains three ConvNeXt-Blocks, each of which contains a depth-wise separable convolution of size 7×7, a dimensionality increase convolution layer of size 1×1, a normalization layer, a GELU activation layer, and a dimensionality reduction convolution layer of size 1×1. The number of output channels is 96. The second convolution group also contains three ConvNeXt-Blocks with an output channel number of 192. The third convolution group contains nine ConvNeXt-Blocks with an output channel number of 384.

[0065] Let the satellite image be I s , the ground panoramic image is I g , I s and I g are input into the first feature extraction module and the second feature extraction module respectively. The first feature extraction module is used to extract the s Perform feature extraction and obtain the size of I s 1 / 16 of the satellite's intermediate characteristic F s The second feature extraction module is used to g Perform feature extraction and obtain the size of I g 1 / 16 of the satellite's intermediate characteristic F g Among them, F s and F g The size of is C×H×W, where C is the number of channels, H is the height, and W is the width.

[0066] In one example, Figure 2 As shown in FIG, the first spatial feature aggregation module and the second spatial feature aggregation module have the same structure, and are both composed of A (generally, the value of A is 8) independent local spatial attention units. Each local spatial attention unit is used to extract significant features of the corresponding intermediate features and keep the number of front and back feature channels unchanged.

[0067] The first spatial feature aggregation module is used to aggregate F s Perform maximum pooling along the channel dimension to obtain the first significant feature mask Then, two fully connected layers are used for feature mixing and selection to obtain the local spatial attention mask. Finally, M s2 With F sThe first spatial feature aggregation module outputs a feature vector f of length C i s , i ∈ [0, A-1].

[0068] The outputs of the local spatial attention units of the first spatial feature aggregation module are spliced in the channel dimension to obtain a satellite global feature f of length AxC s .

[0069] The second spatial feature aggregation module performs maximum pooling along the channel dimension on F g to obtain a second saliency feature mask M After two fully connected layers for feature mixing and selection, a local spatial attention mask M is obtained. g2 Finally, M g is multiplied element by element with F i , and each local spatial attention unit of the second spatial feature aggregation module outputs a feature vector f of length C g , i ∈ [0, A-1].

[0070] The outputs of the local spatial attention units of the second spatial feature aggregation module are spliced in the channel dimension to obtain a satellite global feature f of length AxC g .

[0071] In one example, as shown in FIGS. Figure 2 , Figure 3 , the bird's eye view image transformation module uses the conversion relationship between the spherical coordinate system O sp and the spatial rectangular coordinate system O1X1Y1Z1, the conversion relationship between the normalized equidistant rectangular coordinate system X2O2Y2and the ground pixel coordinate system o p , to derive the conversion relationship between the spherical coordinate system O sp and the normalized equidistant rectangular coordinate system X2O2Y2.

[0072] Suppose there is a point on the spherical coordinate system O sp (as shown in FIG. Figure 3 ), with a default radius of 1, taking the east longitude and north latitude directions as positive directions, denotes the longitude, and θ denotes the latitude, , the value range of θ is [-π / 2, π / 2], and its corresponding coordinate in the spatial rectangular coordinate system is According to the ground panoramic image imaging principle and spherical equidistant projection, the coordinates of in the normalized equidistant rectangular coordinate system O2are (like Figure 3 As shown), then according to The geometric conversion relationship between π and x2 and 1 can be analogized to deduce the spherical coordinate system O sp The conversion relationship with the midpoint coordinates of the normalized rectangular coordinate system X2O2Y2 is expressed by the formula: y2=2θ / π, θ=y2·(π / 2).

[0073] Next, take the position of the upper left corner of the ground panorama as the ground pixel coordinate system o p The origin of the bird's-eye view pixel coordinate system is taken as the position of the upper left corner of the blank bird's-eye view image. b The origin of the spherical coordinate system O sp A section is made at the south pole of the image to serve as the bird's-eye view pixel coordinate system o b The projection plane, according to the bird's-eye view pixel coordinate system o b With the spherical coordinate system O sp The conversion relationship between the spherical coordinate system O sp The conversion relationship between the spatial rectangular coordinate system O1X1Y1Z1 and the spherical coordinate system O sp The transformation relationship between the normalized rectangular coordinate system X2O2Y2 and the spherical coordinate system O sp The next point Corresponding to the point in the normalized rectangular coordinate system X2O2Y2 With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them can be expressed by the formula:

[0074]

[0075] Then, based on the normalized rectangular coordinate system X2O2Y2 With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them is used to derive the ground pixel coordinate system o p The next point With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them can be expressed by the formula:

[0076]

[0077] Among them, the size of the ground panorama is H p ×W p .

[0078] Finally, based on the ground pixel coordinate system o p The next point With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them is used to transform the ground panoramic image into a bird's-eye view image.

[0079] In one example, during the projection process, the projection plane of the bird's-eye view is the South Pole section of the spherical coordinate system, and the maximum value of the projection angle FOV of the ground panorama is controlled to be 85°. After projection, the obtained bird's-eye view is center-cropped to remove the severely distorted parts of the periphery, thereby obtaining a bird's-eye view sub-image with a high similarity to the corresponding area in the satellite image. The center cropping ratio is set to 0.8.

[0080] In one example, Figure 4 As shown in Figure 1, the location-aware prediction module is specifically composed of a feature extractor and a location-aware prediction unit. The location-aware prediction module first uses the feature extractor to extract features from satellite images and bird's-eye view images respectively to obtain satellite prediction features F ps and bird's-eye view prediction feature F pb Next, we use the location-aware prediction unit to ps and F pb , perform position perception prediction, and then use the multi-layer perceptron to predict the matching probability Prob of the image pair pd and the possible center offset delta in the image pair pd , as the location-aware prediction result.

[0081] In one example, the location-aware prediction module uses a parameter-sharing feature extractor to extract global features from perspective-aligned satellite and bird's-eye view image pairs. The first four convolutional modules of VGG16 serve as the primary component of the feature extractor. For satellite images, the output feature map of the third convolutional module of the feature extractor is taken, with a size of 384×16×16. For bird's-eye views, the output feature map of the fourth convolutional module of the feature extractor is taken, with a size of 384×8×8. Similarity is then calculated using the convolutional concept to obtain a 9×9 perceptual heat map M, which is flattened and used as the input to the MLP.

[0082] Step 13: Input the training samples in the training sample set into the cross-view image geolocation model, construct an overall loss function based on the satellite global features, the ground panoramic global features and the position perception prediction results, and use the overall loss function to iteratively train the cross-view image geolocation model. After the training convergence conditions are met, a trained cross-view image geolocation model is obtained.

[0083] In a specific implementation, after the server builds the cross-view image geolocation model, the server can input the training samples in the training sample set into the cross-view image geolocation model, build an overall loss function based on the satellite global feature, the ground panoramic global feature, and the location-aware prediction result, iteratively train the cross-view image geolocation model using the overall loss function, and obtain the trained cross-view image geolocation model after meeting the training convergence condition.

[0084] In one example, in the process of building the overall loss function based on the satellite global feature, the ground panoramic global feature, and the location-aware prediction result, f s and f g are first mapped to the same feature space, the similarity between f s and f g is calculated by distance measurement, and a weighted soft-margin triplet loss function is constructed based on the calculated similarity, so as to shorten the distance between the matched image pairs and lengthen the distance between the unmatched image pairs.

[0085] The weighted soft-margin triplet loss function is represented by the following formula:

[0086] Loss coarse = ln{1+exp[α(d p -d n )]};

[0087] wherein, Loss coarse represents the weighted soft-margin triplet loss function, d p and d n respectively represent the Euclidean distances between the anchor point and the positive sample and the negative sample, and α is a preset hyperparameter.

[0088] Next, a prediction joint loss function is built based on Prob pd and delta pd .

[0089] The prediction joint loss function is represented by the following formula:

[0090]

[0091] wherein, Loss predit represents the prediction joint loss function, B is the batch size, ε is a very small positive number, and delta gt is the center shift true value.

[0092] Finally, an overall loss function Loss total is built based on Loss coarse and Loss predit , and Loss total is represented as:

[0093] Loss total =δ1·Loss coarse +δ2·Loss predit ;

[0094] Among them, δ1 and δ2 are Loss coarse and Loss predit The weight coefficient is δ1+δ2=1.

[0095] In one example, during training, an Adam optimizer may be used for optimization.

[0096] In one example, after obtaining a trained cross-view image geolocation model, the server can input a test image into the trained cross-view image geolocation model, use a dual-branch backbone network for coarse-grained retrieval, and obtain a TopN (generally 20) candidate result set. Then, a fine-grained prediction network is used to predict the matching probability and offset to obtain the top K most similar satellite image query results. The recall rate indicator is used to evaluate the coarse-grained image retrieval results, and the mean error and median error are used to evaluate the fine-grained offset prediction results.

[0097] Step 14: Input the paired satellite images and ground panoramic images to be positioned into the trained cross-view image geolocation model to obtain the positioning results output by the trained cross-view image geolocation model.

[0098] In the specific implementation, after obtaining the trained cross-view image geolocation model, the server can deploy it to the scene where it is needed. When in use, the paired satellite images and ground panoramic images to be located are input into the trained cross-view image geolocation model to obtain the positioning results output by the trained cross-view image geolocation model.

[0099] In one example, Figure 5 As shown in the figure, by querying the GPS latitude and longitude information corresponding to the first K closest satellite images, the task of localizing a ground panoramic image can be completed. On the two VIGOR test sets (Same-Area and Cross-Area), the Recall@1 accuracy of coarse-grained image retrieval was 69.85% and 37.98%, respectively, representing improvements of 4.62% and 1.60% over methods with the same model size.

[0100] The embodiment proposes a space feature aggregation and location-aware progressive cross-view image geolocation method, designs, constructs, trains and uses a cross-view image geolocation model to realize the cross-view image geolocation task. For the center misaligned satellite image and ground panoramic image pair, the task fusion of coarse-grained retrieval matching and fine-grained offset prediction is realized, and the cross-view image geolocation is carried out based on the progressive metric learning idea, which effectively improves the positioning accuracy. The embodiment applies a pseudo-twin trunk network and fine-grained prediction fusion strategy, the double-branch pseudo-twin network trunk is responsible for the coarse-grained reference image candidate set screening, and the fine-grained prediction network realizes the matching degree prediction and center offset estimation by using the bird's eye view image transformation and location-aware prediction. While fusing the two processes, the geolocation ability of the cross-view image geolocation model for the center misaligned image pair is improved, thereby effectively improving the precision and robustness of the cross-view image geolocation.

[0101] The step division of the above various methods is only for the purpose of clear description, and can be combined into one step or some steps can be split and decomposed into multiple steps in implementation, as long as the same logical relationship is included, which is within the protection scope of the present application. The irrelevant modification added in the algorithm or process, or the irrelevant design introduced, but does not change the core design of the algorithm and process, is within the protection scope of the present application.

[0102] Another embodiment of the present application proposes a space feature aggregation and location-aware progressive cross-view image geolocation system. The details of the space feature aggregation and location-aware progressive cross-view image geolocation system proposed in the embodiment will be specifically described below. The following content only provides implementation details for the convenience of understanding, and is not necessary for implementing the embodiment. Figure 6 FIG. 1 is a structural schematic diagram of a space feature aggregation and location-aware progressive cross-view image geolocation system proposed in the embodiment, which includes a training sample set construction module 21, a model construction module 22, a model training module 23 and a model using module 24.

[0103] The training sample set construction module 21 is used to obtain paired satellite images and ground panoramic images as training sample images to construct a training sample set.

[0104] The model construction module 22 is configured to construct a cross-view image geolocation model composed of a double-branch backbone network and a fine-grained prediction network. The two branches of the double-branch backbone network are a satellite processing branch and a ground panoramic processing branch, respectively, for performing global feature extraction on satellite images and ground panoramic images to obtain satellite global features and ground panoramic global features. The fine-grained prediction network is composed of a bird's-eye view image transformation module and a location-aware prediction module. The bird's-eye view image transformation module is configured to perform bird's-eye view transformation on the ground panoramic image based on coordinate system transformation to obtain a bird's-eye view image. The location-aware prediction module is configured to perform location-aware prediction based on the satellite image and the bird's-eye view image to obtain a location-aware prediction result.

[0105] The model training module 23 is configured to input training samples in a training sample set into the cross-view image geolocation model, construct an overall loss function based on the satellite global features, the ground panoramic global features and the location-aware prediction result, and perform iterative training on the cross-view image geolocation model by using the overall loss function. After the training convergence condition is met, a trained cross-view image geolocation model is obtained.

[0106] The model using module 24 is configured to input a pair of satellite images and ground panoramic images to be located into the trained cross-view image geolocation model to obtain a positioning result output by the trained cross-view image geolocation model.

[0107] It can be found that the embodiment is a system embodiment corresponding to the above-mentioned method embodiment, and the embodiment can be implemented in cooperation with the above-mentioned method embodiment. The related technical details and technical effects mentioned in the above-mentioned method embodiment are still valid in the embodiment. In order to reduce repetition, they will not be described here. Correspondingly, the related technical details mentioned in the embodiment can also be applied to the above-mentioned method embodiment.

[0108] It is worth mentioning that each module and module involved in the embodiment is a logical module. In actual application, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units not closely related to solving the technical problems proposed in the present application are not introduced in the embodiment, but this does not mean that there are no other units in the embodiment.

[0109] Another embodiment of the present application proposes an electronic device, the structure of which is as shown in Figure 7As shown, it comprises: at least one processor 31; and a memory 32 connected with the at least one processor 31 in communication; wherein the memory 32 stores instructions executable by the at least one processor 31, and the instructions are executed by the at least one processor 31 to enable the at least one processor 31 to perform a spatial feature aggregation and location-aware progressive cross-view image geolocation method as described in any of the above method embodiments.

[0110] The bus connects one or more processors and the memory, and various circuits of the one or more processors and the memory are connected together by the bus. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and thus will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, which provide a means for communicating with various other apparatuses over a transmission medium. Data processed by the processor is transmitted over the wireless medium via the antenna, and further, the antenna also receives data and transmits the data to the processor.

[0111] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor in performing operations.

[0112] Another embodiment of the present application provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement a spatial feature aggregation and location-aware progressive cross-view image geolocation method as described in any of the above method embodiments.

[0113] That is, those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs are stored in a storage medium, including a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various storage medium that can store program codes.

[0114] Those skilled in the art can understand that the above embodiments are specific embodiments of the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A progressive cross-view image geolocation method based on spatial feature aggregation and location awareness, characterized in that: The method comprises: Obtain paired satellite images and ground panoramic images as sample images to construct a training sample set; A cross-view image geolocation model consisting of a dual-branch backbone network and a fine-grained prediction network is constructed. The two branches of the dual-branch backbone network are the satellite processing branch and the ground panoramic processing branch, which are used to extract global features from satellite images and ground panoramic images, respectively, to obtain satellite global features and ground panoramic global features. The fine-grained prediction network consists of a bird's-eye view image transformation module and a position-aware prediction module. The bird's-eye view image transformation module is used to perform a bird's-eye view transformation on the ground panoramic image based on the transformation of the coordinate system to obtain a bird's-eye view image. The position-aware prediction module is used to perform position-aware prediction based on the satellite image and the bird's-eye view image to obtain a position-aware prediction result. The training samples in the training sample set are input into the cross-view image geolocation model. An overall loss function is constructed based on the satellite global features, the ground panoramic global features, and the position-aware prediction results. The cross-view image geolocation model is iteratively trained using the overall loss function. After the training convergence conditions are met, a trained cross-view image geolocation model is obtained. The paired satellite images and ground panoramic images to be positioned are input into the trained cross-view image geolocation model to obtain the positioning results output by the trained cross-view image geolocation model.

2. The method for progressive cross-view image geolocation based on spatial feature aggregation and location awareness according to claim 1, characterized in that: The satellite processing branch is specifically composed of a first feature extraction module and a first spatial feature aggregation module, and the ground panoramic processing branch is specifically composed of a second feature extraction module and a second spatial feature aggregation module; The structure of the first feature extraction module and the second feature extraction module is the same. Both are composed of the first four groups of convolution modules of ConvNeXt-T. The first four groups of convolution modules of ConvNeXt-T include an initial convolution layer and three consecutive convolution groups. The convolution kernel size of the initial convolution layer is 4×4, the stride is 4, and the number of output channels is 96. The three convolution groups are all composed of ConvNeXt-Block, but the number of blocks in each convolution group is different. The first convolution group contains three ConvNeXt-Blocks. Each block contains a depth-separable convolution of size 7×7, a dimensionality increase convolution layer of size 1×1, a normalization layer, a GELU activation layer, and a dimensionality reduction convolution layer of size 1×1. The number of output channels is 96. The second convolution group also contains three ConvNeXt-Blocks with an output channel number of 192. The third convolution group contains nine ConvNeXt-Blocks with an output channel number of 384. Let the satellite image be I s , the ground panoramic image is recorded as I g , I s and I g are input into the first feature extraction module and the second feature extraction module respectively. The first feature extraction module is used to extract the s Perform feature extraction and obtain the size of I s 1 / 16 of the satellite's intermediate characteristic F s The second feature extraction module is used to g Perform feature extraction and obtain the size of I g 1 / 16 of the satellite's intermediate characteristic F g ; Among them, F s and F g The size of is C×H×W, where C is the number of channels, H is the height, and W is the width.

3. The method for progressive cross-view image geolocation based on spatial feature aggregation and location awareness according to claim 2, characterized in that: The first spatial feature aggregation module and the second spatial feature aggregation module have the same structure, both consisting of A independent local spatial attention units. Each local spatial attention unit is used to extract significant features from the corresponding intermediate features while keeping the number of front and back feature channels unchanged. The first spatial feature aggregation module is used to aggregate F s Perform maximum pooling along the channel dimension to obtain the first significant feature mask Then, two fully connected layers are used for feature mixing and selection to obtain the local spatial attention mask. Finally, M s2 With F s Performing element-by-element multiplication, each local spatial attention unit of the first spatial feature aggregation module will output a feature vector f of length C i s , i∈[0,A-1]; The outputs of each local spatial attention unit of the first spatial feature aggregation module are spliced ​​in the channel dimension to obtain the satellite global feature f with a length of A×C s ; The second spatial feature aggregation module is used to g Perform maximum pooling along the channel dimension to obtain the second salient feature mask Then, two fully connected layers are used for feature mixing and selection to obtain the local spatial attention mask. Finally, M g2 With F g Performing element-by-element multiplication, each local spatial attention unit of the second spatial feature aggregation module will output a feature vector f of length C i g , i∈[0,A-1]; The outputs of each local spatial attention unit of the second spatial feature aggregation module are spliced ​​in the channel dimension to obtain the satellite global feature f with a length of A×C g .

4. The method for progressive cross-view image geolocation based on spatial feature aggregation and location awareness according to claim 3, characterized in that: The bird's-eye view image transformation module is used to perform bird's-eye view transformation on the ground panoramic image based on the transformation of the coordinate system to obtain a bird's-eye view image, including: According to the imaging principle of the ground panorama, using the spherical coordinate system O sp The conversion relationship between the spatial rectangular coordinate system O1X1Y1Z1, the normalized equirectangular coordinate system X2O2Y2 and the ground pixel coordinate system o p The conversion relationship between them leads to the spherical coordinate system O sp The conversion relationship between the normalized rectangular coordinate system X2O2Y2; Take the upper left corner of the ground panorama as the ground pixel coordinate system o p The origin of the bird's-eye view pixel coordinate system is taken as the position of the upper left corner of the blank bird's-eye view image. b The origin of the spherical coordinate system O sp A section is made at the south pole of the image to serve as the bird's-eye view pixel coordinate system o b The projection plane, according to the bird's-eye view pixel coordinate system o b With the spherical coordinate system O sp The conversion relationship between the spherical coordinate system O sp The conversion relationship between the spatial rectangular coordinate system O1X1Y1Z1 and the spherical coordinate system O sp The transformation relationship between the normalized rectangular coordinate system X2O2Y2 and the spherical coordinate system O sp The next point Corresponding to the point in the normalized rectangular coordinate system X2O2Y2 With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them; Points based on the normalized rectangular coordinate system X2O2Y2 With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them is used to derive the ground pixel coordinate system o p The next point With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them; Based on the ground pixel coordinate system p The next point With the bird's-eye view pixel coordinate system o b The projection point below The conversion relationship between them is used to transform the ground panoramic image into a bird's-eye view image.

5. The method for progressive cross-view image geolocation based on spatial feature aggregation and location awareness according to claim 4, characterized in that: The location-aware prediction module consists of a feature extractor and a location-aware prediction unit. The location-aware prediction module is used to perform location-aware prediction based on satellite images and bird's-eye-view images, and obtain location-aware prediction results, including: The feature extractor is used to extract features from satellite images and bird's-eye view images respectively to obtain satellite prediction features F ps and bird's-eye view prediction feature F pb ; Using the location-aware prediction unit, based on F ps and F pb , perform position perception prediction, and then use the multi-layer perceptron to predict the matching probability Prob of the image pair pd and the possible center offset delta in the image pair pd , as the location-aware prediction result.

6. The method for progressive cross-view image geolocation based on spatial feature aggregation and location awareness according to claim 5, characterized in that: The overall loss function is constructed based on the satellite global features, ground panoramic global features and location-aware prediction results, including: f s and f g Mapped into the same feature space, f is calculated by distance metric s With f g The similarity between them is calculated, and a weighted soft margin triplet loss function is constructed based on the calculated similarity to shorten the distance between matching image pairs and increase the distance between mismatching image pairs. The weighted soft margin triplet loss function is expressed as: Loss coarse =ln{1+exp[α(d p -d n )]}; Among them, Loss coarse represents the weighted soft margin triplet loss function, d p and d n Represents the Euclidean distance between the anchor point and the positive sample and the negative sample respectively, and α is a preset hyperparameter; Based on Prob pd and delta pd Construct prediction joint loss function; The prediction joint loss function is expressed as: Among them, Loss predit Represents the prediction joint loss function, B is the batch size, ε is a very small positive number, delta gt is the true value of the center offset; Based on Loss coarse and Loss predit Constructing the overall loss function Loss total , Loss total Expressed as: Loss total =δ1·Loss coarse +δ2·Loss predit ; Among them, δ1 and δ2 are Loss coarse and Loss predit The weight coefficient is δ1+δ2=1.

7. A method for progressive cross-view image geolocation based on spatial feature aggregation and location awareness according to any one of claims 1 to 6, characterized in that: After obtaining the trained cross-view image geolocation model, the method further includes: The test image is input into the trained cross-view image geolocation model, and a two-branch backbone network is used for coarse-grained retrieval to obtain the TopN candidate result set. The matching probability and offset are then predicted through a fine-grained prediction network to obtain the top K closest satellite image query results. The recall rate indicator is used to evaluate the coarse-grained image retrieval results, and the mean error and median error are used to evaluate the fine-grained offset prediction results.

8. A progressive cross-view image geolocation system based on spatial feature aggregation and location awareness, characterized by: The system comprises: A training sample set construction module is used to obtain paired satellite images and ground panoramic images as training sample images to construct a training sample set; A model construction module is used to construct a cross-view image geolocation model consisting of a dual-branch backbone network and a fine-grained prediction network. The two branches of the dual-branch backbone network are the satellite processing branch and the ground panoramic processing branch, which are used to extract global features from satellite images and ground panoramic images, respectively, to obtain satellite global features and ground panoramic global features. The fine-grained prediction network consists of a bird's-eye view image transformation module and a position-aware prediction module. The bird's-eye view image transformation module is used to perform a bird's-eye view transformation on the ground panoramic image based on the transformation of the coordinate system to obtain a bird's-eye view image. The position-aware prediction module is used to perform position-aware prediction based on the satellite image and the bird's-eye view image to obtain a position-aware prediction result. A model training module is used to input training samples from the training sample set into the cross-view image geolocation model, construct an overall loss function based on satellite global features, ground panoramic global features, and position-aware prediction results, and iteratively train the cross-view image geolocation model using the overall loss function. After the training convergence conditions are met, a trained cross-view image geolocation model is obtained; The model usage module is used to input the paired satellite images and ground panoramic images to be located into the trained cross-view image geolocation model to obtain the positioning results output by the trained cross-view image geolocation model.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a progressive cross-view image geolocation method based on spatial feature aggregation and location awareness as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement a spatial feature aggregation and location-aware progressive cross-view image geolocation method according to any one of claims 1 to 7.

Images