A geolocation method based on ground-air cross view

Abstract

This invention discloses a ground-to-air cross-view geolocation method, comprising the following steps: S1: establishing a ground-to-air image training set; S2: performing regional segmentation on the ground image; S3: converting the overhead view of the satellite image into a frontal view; S4: training a two-layer conditional generative adversarial network to obtain satellite images with a ground-view style; S5: training a Transformer network; and S6: obtaining the latest geolocated ground and satellite images and inputting them into the trained Transformer network for image matching. This invention proposes, for the first time, a method for automatically hybridizing perspective-polar coordinate mapping and a two-layer conditional generative adversarial network to reduce the visual domain spacing of ground-to-air images. This method combines image alignment with the Transformer method for the first time, achieving accurate cross-view geolocation.

Description

Technical Field

[0001] The present invention belongs to the field of geographic positioning technology, and in particular relates to a ground-air cross-view geographic positioning method. Background Art

[0002] Cross-view image matching is the process of retrieving the most relevant images from different platforms. Cross-view geographic localization (CVGL) is mainly based on ground imagery (front view) and satellite imagery (top view). The goal of ground-to-air cross-view geolocalization is to determine the location of a given ground image by matching it with satellite imagery. However, due to the large difference in viewpoints between ground-to-air images, this remains a very challenging task.

[0003] With the rapid development of convolutional neural networks (CNNs) in computer vision, existing methods are primarily based on CNNs. Early research focused directly on feature representation. Metric learning methods that directly focus on features have gradually been applied to CVGL. These methods directly extract view-invariant features without performing explicit view transformations on the input image. They only learn feature representations based on image content and do not consider the spatial correspondence between image pairs. Hu et al., in "Sixing Hu, Mengdan Feng, Rang MHNguyen, and Gim Hee Lee, “Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization,” in 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2018, pp. 7258–7267," used a global VLAD descriptor and proposed CVM-Net, a Siamese architecture with a weighted soft-edge loss. Liu and Li were the first to use orientation information as an important cue for spatial localization tasks in the paper “Liu Liu and Hongdong Li, “Lending orientation to neural networks for cross-view geo-localization,” in 2019 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019, pp. 5624–5633.” Shi et al. considered domain differences and spatial layout information in the paper “Yujiao Shi, Xin Yu, Liu Liu, Tong Zhang, and Hongdong Li, “Optimal feature transport for cross-view image geo-localization,” in The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI, 2020, pp. 11990–11997.” and proposed a feature transfer network (CVFT) that takes into account domain differences and spatial layout information.

[0004] Later studies considered the spatial correspondence between ground-air image pairs and solved the visual domain spacing problem by transforming the input image, such as performing explicit viewpoint mapping on the input image and using conditional generative adversarial networks (CGANs). Shi et al., in the paper "Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li, "Spatial-aware feature aggregation for image based cross-view geo-localization," in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS, 2019, pp. 10090–10100," used the explicit viewpoint mapping and proposed SAFA, which uses polar coordinate mapping to eliminate differences in the geometric domain and proposes a spatial perception layer. Toker et al. used a conditional generative adversarial network (CGN) in the paper “Aysim Toker, Qunjie Zhou, Maxim Maximov, and Laura Leal-Taix′e, “Coming down to earth: Satellite-to-street view synthesis for geo-localization,” in 2021IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2021, pp. 6488–6497.” to create realistic ground images from satellite imagery and simultaneously localize the corresponding ground view queries in an end-to-end manner. A key method to reduce the visual domain gap between different views is viewpoint mapping, but existing mapping methods all use the same mapping for the entire image, ignoring the special spatial correspondence between the common view and non-common view regions of the image. In addition, existing CGAN methods do not utilize additional effective information.

[0005] Transformer has achieved excellent results in natural language processing (NLP). Its parallel computing, global vision, and flexible stacking capabilities are unmatched by CNN. Recent research has attempted to use it for computer vision, but its application in CVGL is still very limited. Yang et al. proposed a simple and effective self-cross attention mechanism Polar-EgoTR in the paper "Hongji Yang, Xiufan Lu, and Yingying Zhu, "Cross-view geo-localization with evolving transformer," CoRR, vol.abs / 2107.00842, 2021." to improve the quality of learned representations. Existing methods do not preprocess the image before the Transformer, ignoring the direct spatial correspondence between image pairs.

[0006] Furthermore, existing methods fail to consider that the corresponding scene in an image pair is not captured simultaneously and may even change over time, such as due to changes in vegetation and buildings. Only the road remains unchanged with high probability, playing a crucial role in matching. Therefore, semantic segmentation techniques are used to segment different regions and augment the data to fully utilize road information. Summary of the Invention

[0007] In order to solve the above problems, the present invention proposes a ground-air cross-view geolocation method.

[0008] The technical solution of the present invention is: a ground-air cross-view geolocation method comprising the following steps:

[0009] S1: Acquire geolocated ground images and satellite images to build a ground-air image training set;

[0010] S2: Use semantic segmentation method to perform regional segmentation on ground images;

[0011] S3: Convert the top view of the satellite image into the front view by hybrid perspective mapping and polar coordinate mapping;

[0012] S4: The segmented ground image and the converted satellite image are used as inputs to a two-layer conditional generative adversarial network. The two-layer conditional generative adversarial network is trained to obtain satellite images with a ground view style.

[0013] S5: Input the segmented ground image and the satellite image with ground view style into the Transformer network for matching, completing the Transformer network training;

[0014] S6: Obtain the latest geolocated ground images and satellite images and input them into the trained Transformer network for image matching to complete ground-air cross-view geolocation.

[0015] Semantic segmentation is used to segment different areas, fully utilizing road information and augmenting the data. Since segmented images often contain useful information, the purpose of low-level visual features is to generate more effective image descriptors for better image feature retrieval performance. Therefore, the key to using semantic segmentation is using the segmented images as input to the subsequent DCGAN (Dual Conditional Generative Adversarial Nets) algorithm. This leverages this additional useful information and generates satellite imagery with a more ground-level aesthetic.

[0016] Furthermore, in step S2, the calculation formula for perspective mapping is:

[0017]

[0018] Among them, x a Represents the horizontal coordinate of the original satellite image used for perspective mapping, y a Indicates the vertical coordinate of the original satellite image used for perspective mapping, x g Represents the horizontal coordinate of the satellite image after perspective mapping transformation, y g represents the vertical coordinate of the satellite image after perspective mapping transformation, C h Indicates the height of the camera from the ground plane, H a Represents the height of the satellite image after perspective mapping, W a Indicates the width of the satellite image after perspective mapping;

[0019] The calculation formula for polar coordinate mapping is:

[0020]

[0021] Among them, x' a Represents the horizontal coordinate of the original satellite image used for polar coordinate mapping, y' a Indicates the vertical coordinate of the original satellite image used for polar coordinate mapping, x' g Represents the horizontal coordinate of the satellite image after polar coordinate mapping conversion, y' g A represents the vertical coordinate of the satellite image after polar coordinate mapping conversion. a Indicates the size of the original satellite image, H g Indicates the height of the satellite image after polar coordinate mapping, W g Indicates the width of the satellite image after polar coordinate mapping.

[0022] The beneficial effect of the above further solution is that, while deep neural networks can theoretically learn any functional transformation, the learning process can be very burdensome. This invention explicitly aligns the two domains based on geometric correspondence, promoting network convergence and reducing the learning burden. Rather than forcing the neural network to learn an implicit mapping, the satellite imagery is explicitly transformed, converting the top view of the satellite imagery into a front view through an automatic hybrid polar-perspective mapping, resulting in approximate alignment with the ground imagery. This establishes a better spatial correspondence, roughly bridging the gap in geometric spatial correspondence between the two domains.

[0023] Cross-view image pairs have common view regions and non-common view regions. In the non-common view region, only one side of vertical structures can be seen in the top-down or ground view, but planar structures can be seen simultaneously in the common view region. Therefore, their geometric spatial correspondence should not be achieved using the same method. Our proposed hybrid polar coordinate-perspective mapping method utilizes perspective mapping and polar coordinate mapping, respectively, based on the essential differences between common view and non-common view regions. This mapped satellite image closely matches the actual ground image, making it more suitable for matching ground panoramic images with satellite images, significantly reducing the visual domain gap between ground and air images.

[0024] Furthermore, in step S4, the loss function of the second generator G2 in the two-layer conditional generative adversarial network is expressed as:

[0025] S' g =G1(I a ),I' g =G2(S' g )

[0026]

[0027] Among them, S' g Represents the image generated by the first generator, I' g Represents the image generated by the second generator, I a represents the input hybrid perspective-polar mapped satellite image, S g Represents the semantic segmentation map of the input ground image, I g represents the input original ground image, L CGAN (·) represents the CGAN loss function, L L1 (·) represents the L1 loss function, G1 and G2 represent the first and second generators, D2 represents the second discriminator, E[·] represents the expectation function between the two, log represents the logarithmic function, D2[·] represents the variance function between the two, and ||·||1 represents the 1-norm.

[0028] Furthermore, in step S4, the overall objective function L of the two-layer conditional generative adversarial network isDual-CGAN The expression is:

[0029] L Dual-CGAN =L CGAN (G1,D1)+λL L1 (G1)+L CGAN (G2,D2)+λL L1 (G2)

[0030] Among them, L CGAN (·) represents the CGAN loss function, L L1 (*) denotes the L1 loss function, G1 denotes the first generator, G2 denotes the second generator, D1 denotes the first discriminator, D2 denotes the second discriminator, and λ denotes the loss function L L1 The weight parameter of .

[0031] The beneficial effect of the above further solution is that: in the invention, the polar coordinate-perspective mapping does not take into account the scene content, the actual correspondence between the two different domains is much more complicated than a simple mapping, and the appearance distortion of the transformed image is still very obvious. This alone is not enough to completely eliminate the domain gap between the two views. To solve this problem, the present invention synthesizes ground images with realistic appearance and content preservation from the corresponding satellite views to address the huge perspective difference between the two domains in terms of geolocation. The present invention adopts the DCGAN method, the goal of which is to take the converted satellite image and the semantically segmented ground image as input, and simultaneously train a two-layer conditional generative adversarial network to synthesize satellite images with the style of the ground view.

[0032] Furthermore, step S5 includes the following sub-steps:

[0033] S51: Segment the ground image after region segmentation and the satellite image with ground view style into several patches, and use the linear patch embedding as the input of the Transformer network;

[0034] S52: embed position into x pos and learnable class embedding x class Add to the input of the Transformer network to get the input sequence embedding x0;

[0035] S53: Embed the input sequence x0 into the L-layer Transformer encoder of the Transformer network for image matching.

[0036] Furthermore, in step S52, the calculation formula for embedding the input sequence x0 is:

[0037] x0=[x class ;x]+x pos

[0038] Among them, x pos represents the position embedding, x class represents a learnable class embedding.

[0039] The beneficial effect of the above further scheme is that in the invention, the parallel computing, global vision and flexible stacking capabilities of Transformer are unmatched by CNN. By observing the CNN-based methods in CVGL, two potential problems are found. On the one hand, CVGL needs to mine relevant information between contexts. Images from different domains have position transformations, such as rotation, scaling and offset. Therefore, it is necessary to fully understand the semantic information of the global context. On the other hand, fine-grained information is very important for retrieval tasks. The downsampling operations of CNN-based methods, namely pooling and strided convolution, will reduce the resolution of the image and invisibly destroy the recognizable fine-grained information. In view of this, Transformer will play a role in CVGL as a powerful context-sensitive information extractor. However, its application in CVGL is still very limited, and the existing methods ignore the direct spatial correspondence of images and do not pre-process the images. Therefore, a Transformer-based CVGL model SMDT (the first letters of steps S2-S5, respectively) is proposed. This is the first model that combines image mapping with Transformer.

[0040] The beneficial effects of the present invention are as follows: the present invention proposes for the first time an automatic hybrid perspective-polar coordinate mapping method to reduce the visual domain spacing of ground-to-air images; proposes for the first time a two-layer conditional generative adversarial network that takes into account additional effective information by taking the transformed image and the semantically segmented image as input; proposes a Transformer-based CVGL model SMDT, which is the first model to combine image mapping with Transformer; and utilizes semantic segmentation technology to promote the segmentation of common view and non-common view areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of the cross-view geolocation method;

[0042] Figure 2 Schematic diagram of semantic segmentation used to obtain class-specific masks corresponding to the ground image (gray);

[0043] Figure 3 Map the effects of satellite images using different methods;

[0044] Figure 4 This is the network architecture diagram of DCGAN;

[0045] Figure 5This is the overall framework diagram of the Transformer method;

[0046] Figure 6 This is a diagram of the Transformer internal method framework. DETAILED DESCRIPTION

[0047] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0048] like Figure 1 As shown, the present invention provides a ground-air cross-view geolocation method, comprising the following steps:

[0049] S1: Acquire geolocated ground images and satellite images to build a ground-air image training set;

[0050] S2: Use semantic segmentation method to perform regional segmentation on ground images;

[0051] S3: Convert the top view of the satellite image into the front view by hybrid perspective mapping and polar coordinate mapping;

[0052] S4: The segmented ground image and the converted satellite image are used as inputs to a two-layer conditional generative adversarial network. The two-layer conditional generative adversarial network is trained to obtain satellite images with a ground view style.

[0053] S5: Input the segmented ground image and the satellite image with ground view style into the Transformer network for matching, completing the Transformer network training;

[0054] S6: Obtain the latest geolocated ground images and satellite images and input them into the trained Transformer network for image matching to complete ground-air cross-view geolocation.

[0055] In an embodiment of the present invention, in step S1, semantic segmentation is performed on the ground image to achieve segmentation of common view areas and non-common view areas, thereby enhancing image sample data. In the subsequent S3 hybrid polar coordinate-perspective mapping, the added samples and other additional information are used to better distinguish between common view areas and non-common view areas, so as to select different mapping methods for different areas. In addition, the keep road mode can also make better use of the additional information of the road, because the image pairs are taken at different times, and buildings and trees may not be used as features for image matching. Only the road is likely to remain unchanged and can be matched as a landmark feature, which plays an important role in image matching. The most important thing is that the semantic segmentation map is used as a conditional input in DCGAN in the subsequent S4, making full use of the additional additional effective information and using more effective image descriptors to achieve better image feature retrieval performance.

[0056] like Figure 2 As shown in , the semantic segmentation module is used to obtain the segmented image corresponding to the ground view. The classes considered are sky, road, sidewalk, tree, building and car, as shown in Figure 2 As shown in part A of , masks are used to generate and distinguish these different classes so that they can be used to create enhanced samples in keep and remove modes, which are two different image manipulation methods. The original pixels of a specific category are retained in keep mode, such as Figure 2 As shown in part B of the figure, the keep mode retains some areas of the original image, while when operating in the remove mode, they are masked by black pixels, as shown in Figure 2 As shown in Section C of Figure 3, remove mode removes parts of the original image. All object classes are considered in remove mode, but sky and car are not considered in keep mode because regions in these classes are not available for matching in the bird's-eye view. Modifying the ground view does not modify the satellite view.

[0057] In the embodiment of the present invention, in step S2, formally, let A a ×A a Indicates the size of the satellite image, H g ×W g represents the target size of polar coordinate transformation. Therefore, the original satellite image point (x' a ,y' a ) and the target transformed satellite image point (x' g ,y' g ) is defined as:

[0058]

[0059] Among them, x' a Represents the horizontal coordinate of the original satellite image used for polar coordinate mapping, y' a Indicates the vertical coordinate of the original satellite image used for polar coordinate mapping, x' g Represents the horizontal coordinate of the satellite image after polar coordinate mapping conversion, y' g A represents the vertical coordinate of the satellite image after polar coordinate mapping conversion. a Indicates the size of the original satellite image, H g Indicates the height of the satellite image after polar coordinate mapping, W g Indicates the width of the satellite image after polar coordinate mapping.

[0060] Align the ground panoramic image with the satellite image and establish a spatial correspondence for the cross-view image pairs. The ground-air image training set containing the ground image (front view) and the satellite image (top view) established in step S1 is used. Specifically, the satellite image (top view) is generated by orthogonal projection, and the ground panoramic image (front view) is generated by projection spherical rectangular projection. Therefore, there are common view areas and non-common view areas in the cross-view image pairs. The common view areas are mainly planar structures that can be seen simultaneously on the ground and satellite views, such as roads and sidewalks. This means that the pixels belonging to the common view areas in the cross-view image pairs are associated through perspective mapping. In contrast, the non-common view areas are composed of vertical structures, such as building roofs and tree crowns. Only one side of these vertical structures can be seen in the top-down view or the ground plane view. In the non-common view area, there is only a semantic relationship between the cross-view image pairs. The present invention proposes a hybrid polar coordinate-perspective mapping method: according to the essential differences between the common view area and the non-common view area, perspective mapping and polar coordinate mapping are respectively adopted. From Figure 3 As can be seen from part (c) of the figure, the satellite image with hybrid perspective-polar coordinate mapping is very close to the actual ground image and is more suitable for matching, which greatly reduces the visual domain gap between the ground and air images.

[0061] For the common view area, a perspective transformation is used to obtain a satellite image that is very similar to the ground image. For a given pixel (x a ,y a ) and the corresponding pixel in the ground image coordinate system (x g ,y g ), the mapping relationship is described by the following equation:

[0062]

[0063] Among them, x a Represents the horizontal coordinate of the original satellite image used for perspective mapping, y a Indicates the vertical coordinate of the original satellite image used for perspective mapping, x g Represents the horizontal coordinate of the satellite image after perspective mapping transformation, y g represents the vertical coordinate of the satellite image after perspective mapping transformation, C h Indicates the height of the camera from the ground plane, H a Represents the height of the satellite image after perspective mapping, W a Indicates the width of the perspective mapped satellite image.

[0064] However, the fundamental assumption of perspective mapping is that all objects represented by pixels within the common view region lie on the same plane. This assumption is not true for vertical structures within non-common view regions. In fact, these vertical objects produce significant distortion, rendering them inaccessible even to terrestrial imagery. Therefore, polar coordinate mapping is used. By applying polar coordinate mapping, the gap in the projected geometry between terrestrial and satellite imagery is roughly bridged.

[0065] In the embodiment of the present invention, in step S4, mapping the satellite image to the ground image using CGAN can be regarded as a process of visual domain adaptation. Krishna Regmi proposed the X-Fork and X-Seq structures to assist in cross-view conversion of images with the help of CGAN. The model of the present invention is mainly based on the improvement of X-Fork and X-Seq, such as Figure 4 As shown, the present invention proposes a DCGAN, in which the first network generates a cross-view segmentation image, and the second network takes the segmentation image from the first generator as input to generate a ground image. The entire system is trained end-to-end, so the two CGANs are trained simultaneously. Compared with the X-Seq framework, the semantic segmentation map of the ground image is added to G1 to participate in the training. Because the semantic segmentation map is used to generate more details based on the contour information of the image, the purpose of the low-level visual features is to generate more effective image descriptors to achieve better image feature retrieval performance. Although more details often lead to a certain amount of confusion in the image, extended experiments have proved that it is worth the trade-off.

[0066] Referring to the loss function of the traditional CGAN, an equivalent expression for the cross-view CGAN network loss in this architecture is obtained. G2 regards the ground image generated by G1 as a conditional input, and the loss function expression of the two-layer conditional generative adversarial network is:

[0067] S' g =G1(I a ),I' g =G2(S' g )

[0068]

[0069] Among them, S' g Represents the image generated by the first generator, I' g represents the image generated by the second generator (i.e., a satellite image with a ground view style), I a represents the input hybrid perspective-polar mapped satellite image, S g Represents the semantic segmentation map of the input ground image, I g represents the input original ground image, L CGAN (·) represents the CGAN loss function, LL1 (·) represents the L1 loss function, G1 and G2 represent the first and second generators, D2 represents the second discriminator, E[·] represents the expectation function between the two, log is the logarithmic function, D2[·] represents the variance function between the two, and ||·||1 is the 1 norm (i.e., satellite images with ground view style and pixel-by-pixel differences with ground images).

[0070] In the embodiment of the present invention, in step S4, the overall objective function L of the two-layer conditional generative adversarial network is Dual-CGAN The expression is:

[0071] L Dual-CGAN =L CGAN (G1,D1)+λL L1 (G1)+L CGAN (G2,D2)+λL L1 (G2)

[0072] Among them, L CGAN (·) represents the CGAN loss function, L L1 (*) denotes the L1 loss function, G1 denotes the first generator, G2 denotes the second generator, D1 denotes the first discriminator, D2 denotes the second discriminator, and λ denotes the loss function L L1 The weight parameter of .

[0073] In this embodiment of the present invention, step S5 includes the following sub-steps:

[0074] S51: Segment the ground image after region segmentation and the satellite image with ground view style into several patches, and use the linear patch embedding as the input of the Transformer network;

[0075] S52: embed position into x pos and learnable class embedding x class Add to the input of the Transformer network to get the input sequence embedding x0;

[0076] S53: Embed the input sequence x0 into the L-layer Transformer encoder of the Transformer network for image matching.

[0077] In this embodiment of the present invention, in step S52, the calculation formula for embedding the input sequence x0 is:

[0078] x0=[x class ;x]+x pos

[0079] Among them, x pos represents the position embedding, x class represents a learnable class embedding.

[0080] The global context-aware feature of SDMT effectively reduces the visual domain spacing, while the position encoding gives it a geometric concept, thereby reducing the ambiguity caused by geometric misalignment. This paper follows the method of applying Transformer to CVGL by Yang et al., but does not use their self-cross attention mechanism, and directly adopts the transformer layer structure in ViT. The specific structure is as follows Figure 5 and Figure 6 shown.

[0081] Vision Transformer: First, the Vision Transformer (ViT) architecture is described as follows. Figure 6 As shown in Figure 2, given an input image, ViT first divides the image into several patches. Then, ViT receives a sequence of linearly projected patch embeddings as input. Where N is the number of patches and D is the patch embedding size. After pre-preparing a learnable class, embed The state at the output of ViT is the image representation, and the position is embedded in x pos Add to X, and get x0 = [x class ;x]+x pos , and feed it into an L-layer Transformer encoder. Each layer consists of a Multihead Self-Attention module (SA) and LayerNorm blocks (LN). Note that MSA consists of multiple self-attention heads and a linear projection block.

[0082] Domain-specific Transformer: The huge regional differences between ground and satellite images indicate that it is difficult to match ground and satellite images in the same data space. To adapt to the cross-view geolocation task, a domain-specific Siamese-style structure is adopted with two independent ViT branches of the same structure to learn ground and satellite image representations respectively. The method framework is as follows Figure 5 ,Each branch is a hybrid structure consisting of a ResNet backbone that extracts CNN feature maps from the image input and a ViT that models the global context from the CNN feature maps.,By treating each 1×1 feature as a patch, a linear projection of the patch embedding in ViT is applied to the CNN feature map.

[0083] Learnable location embeddings: Geometric cues can greatly simplify the cross-view geolocation task. This paper adopts an efficient and flexible method to endow the network with geometric concepts instead of imposing predefined direction knowledge on the network. Specifically, a learnable one-dimensional location embedding is used in ViT, i.e. By adding positional embeddings to linear patch embeddings, the transformed features become position-dependent. Furthermore, since no assumptions are made about positional knowledge and instead learning is done through a learning objective, the proposed SDMT has broader practical applicability. Experiments show that incorporating learnable positional embeddings helps capture relative position information, which is more applicable to images with unknown orientations than absolute position information. Furthermore, SDMT considers scene content when correlating cross-view geometry, which complements the polar-perspective transform and leads to better localization performance.

[0084] The following is an explanation with reference to specific embodiments. The present invention proposes a cross-view image matching method SMDT, which combines cross-view synthesis with geolocation. SMDT fully considers the variability of different regions of the scene in the ground-air image pair, the particularity of the spatial correspondence of the common view-non-common view regions, the effectiveness of additional information, and the limited application of Transformer in CVGL, which are ignored in existing methods, and achieves new state-of-the-art performance with the advantage of high matching accuracy, as shown in Tables 1 and 2. Table 1 shows the comparison with other methods on the CVUSA dataset, and Table 2 shows the comparison with other methods on the CVACT dataset.

[0085] Table 1

[0086] Method R@1 R@5 R@10 R@Top1 CVM-Net 22.47 49.98 63.18 93.62 Liu and Li 40.79 66.82 76.36 96.12 CVFT 61.43 84.69 90.49 99.02 SAFA 81.15 94.23 96.85 99.49 Toker et al. 92.56 97.55 98.33 99.67 Polar-Ego TR 94.05 98.27 98.99 99.67 SMDT (present invention) 95.02 98.97 99.25 99.87

[0087] Table 2

[0088] Method R@1 R@5 R@10 R@Top1 CVM-Net 20.15 45.00 56.87 87.57 Liu and Li 46.96 68.28 75.48 92.01 CVFT 61.05 81.33 86.52 95.93 SAFA 78.28 91.60 93.79 98.15 Toker et al. 83.28 93.57 95.42 98.22 Polar-Ego TR 84.89 94.59 95.96 98.37 SMDT (present invention) 85.52 94.97 96.28 98.96

[0089] The working principle and process of the present invention are as follows: The present invention proposes a cross-view geolocation method called SMDT. First, semantic segmentation technology is used to segment different areas of the ground image; then, the overhead view of the satellite image is converted into a front view by automatically mixing perspective-polar coordinate mapping, which greatly reduces the visual domain distance of the ground-air image pair; then, a DCGAN algorithm is proposed, which takes the converted image and the semantically segmented image as input, and simultaneously trains a two-layer conditional generative adversarial network to synthesize satellite images with a ground view style; finally, the Transformer is used to explicitly use the self-attention property to model global dependencies. The problems of variability of different areas of the scene in the ground-air image pair, which are ignored in existing methods, the spatial correspondence specificity of the common view-non-common view area, the effectiveness of additional information, and the limited application of Transformer in CVGL are respectively solved.

[0090] The beneficial effects of the present invention are as follows: the present invention proposes for the first time an automatic hybrid perspective-polar coordinate mapping method to reduce the visual domain spacing of ground-to-air images; proposes for the first time a two-layer conditional generative adversarial network that takes into account additional effective information by taking the transformed image and the semantically segmented image as input; proposes a Transformer-based CVGL model SMDT, which is the first model that combines image mapping with Transformer; and utilizes semantic segmentation technology to promote the segmentation of common view and non-common view areas.

[0091] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A ground-air cross-view geolocation method, characterized in that: The following steps are involved: S1: Acquire geolocated ground images and satellite images to build a ground-air image training set; S2: Use semantic segmentation method to perform regional segmentation on ground images; S3: Convert the top view of the satellite image into the front view by hybrid perspective mapping and polar coordinate mapping; S4: The segmented ground image and the converted satellite image are used as inputs to a two-layer conditional generative adversarial network. The two-layer conditional generative adversarial network is trained to obtain satellite images with a ground view style. S5: Input the segmented ground image and the satellite image with ground view style into the Transformer network for matching, completing the Transformer network training; S6: Obtain the latest geolocated ground and satellite images and input them into the trained Transformer network for image matching to complete ground-air cross-view geolocation; In step S4, in the two-layer conditional generative adversarial network, the first generator For generating cross-view segmentation maps, the second generator For the first generator Get the segmented image as input and generate the ground image; The first generator and the second generator Train simultaneously, and in the first generator Added semantic segmentation map of ground image Participate in training; The second generator in the two-layer conditional generative adversarial network G The expression of the loss function of 2 is: in, represents the image generated by the first generator, represents the image generated by the second generator, represents the input hybrid perspective-polar mapped satellite image, represents the input original ground image, represents the CGAN loss function, represents the L1 loss function, represents the first generator, represents the second generator, represents the second discriminator, The input is the original ground image And the image generated by the second generator , the conditional variable is the image generated by the first generator , represents the expected function, represents the logarithmic function, is 1 norm; In step S4, the overall objective function of the two-layer conditional generative adversarial network is The expression is: in, represents the first discriminator, whose input is the semantic segmentation map of the ground image And the image generated by the first generator , the conditional variable is the input hybrid perspective-polar coordinate mapping satellite image , Expressed as a loss function The weight parameter of The step S5 includes the following sub-steps: S51: Segment the ground image after region segmentation and the satellite image with ground view style into several patches, and use the linear patch embedding as the input of the Transformer network; S52: Embedding the location and learnable class embeddings Add to the input of the Transformer network to get the input sequence embedding ; S53: Embed the input sequence Input to the L-layer Transformer encoder of the Transformer network for image matching; In step S52, the input sequence is embedded The calculation formula is: in, represents position embedding, represents the learnable class embedding and x represents the input embedding matrix.

2. The ground-air cross-view geolocation method according to claim 1, characterized in that: In step S3, the calculation formula for perspective mapping is: in, represents the horizontal coordinate of the original satellite image used for perspective mapping, represents the vertical coordinate of the original satellite image used for perspective mapping, Represents the horizontal coordinate of the satellite image after perspective mapping transformation, Represents the vertical coordinate of the satellite image after perspective mapping transformation, Indicates the height of the camera from the ground plane, Indicates the height of the satellite image after perspective mapping, Indicates the width of the satellite image after perspective mapping; The calculation formula for polar coordinate mapping is: in, Indicates the horizontal coordinate of the original satellite image used for polar coordinate mapping, Indicates the vertical coordinate of the original satellite image used for polar mapping, Indicates the horizontal coordinate of the satellite image after polar coordinate mapping conversion, Indicates the vertical coordinate of the satellite image after polar coordinate mapping conversion, Indicates the size of the original satellite image, Indicates the height of the satellite image after polar coordinate mapping. Indicates the width of the satellite image after polar coordinate mapping.

Images