Geometric relationship constraint-based cross-view image geographic positioning method and system

Abstract

The invention discloses a cross-view-angle image geographic positioning method and system based on geometrical relationship constraint, and the method comprises the steps: 1, collecting a ground panoramic image and a satellite image, and converting the ground panoramic image into a bird's-eye view angle image; 2, inputting the bird's-eye view angle image and the satellite image into a CNN (Convolutional Neural Network) and Transform combined feature extraction framework; step 3, training a feature extraction framework of the CNN combined with Transform; 4, constructing a satellite image global feature descriptor database; and 5, inputting a to-be-queried ground panoramic image into the trained CNN combined Transform feature extraction framework and the global feature aggregator in sequence, and matching a to-be-queried aerial view image global descriptor with a satellite image global descriptor in a satellite image global feature descriptor database. According to the method, the geometric mapping relation between the ground panoramic image and the aerial view angle image is determined, and the semantic segmentation network and the feature extraction framework are constructed, so that the ground panoramic image to be queried is positioned.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-view image geolocation, and in particular to a cross-view image geolocation method and system based on geometric relationship constraints. Background Art

[0002] Cross-view image geolocation refers to a technology that matches the ground view image (side view) to be queried with the aerial or satellite image (aerial view) with location information to determine the location information of the ground view image. This technology has important application value in the fields of remote sensing information utilization, unmanned driving and augmented reality. Because the ground view image and the aerial view image are collected from different perspectives of different sensors, the drastic difference in perspective makes the externalization of the same scene content in the two images completely different, and the traditional manual feature extraction operator fails because it cannot effectively cope with such scale and content changes. In recent years, deep learning technology has achieved great success in the field of image processing with its powerful feature extraction capabilities, and has also promoted the research of cross-view image geolocation to make new progress.

[0003] At present, most of the cross-view image geolocation algorithms based on deep learning choose to transform this task into an image retrieval problem. The specific form is to extract features from ground and aerial images through learnable neural networks, and use metric learning methods to bring matching features closer and alienate non-matching features, so as to achieve the purpose of image retrieval and positioning. Therefore, the key to solving this problem is how to make up for the image domain gap between aerial and ground images and establish feature associations between images of different views. Shi et al. proposed a cross-view feature transfer module CVFT, which guides the feature matching between street view images and satellite images by realizing the mapping transformation between the feature domains of matching images, thereby effectively reducing the impact of perspective differences. In order to strengthen the generation of global descriptors, Hu et al. added the NetVLAD layer to the network framework, and in order to solve the disadvantage of slow convergence of soft margin triplet loss, they proposed a weighted soft margin triplet loss, in which the distance between positive and negative samples is additionally scaled. Shi et al.

[12] preprocessed satellite images with polar coordinate transformation. The resulting transformed images achieved basic image domain alignment with ground panoramic images, greatly reducing the difficulty of cross-view image matching and providing a new solution for subsequent cross-view image geolocation tasks. Although the polar coordinate transformation method can eliminate the image domain gap to the greatest extent, it will introduce image distortion into the transformed image, thus forming interference. In order to solve this interference, Toker et al. used the powerful image generation capability of generative adversarial networks (GANs) to process polar coordinate transformed images, thereby obtaining more realistic synthetic street scene images, and based on this, they better completed the cross-view image geolocation task. Although the above algorithms have made certain breakthroughs, whether directly using the network to extract features or using preprocessing operations to transform images, they are overly dependent on image content and ignore the mining of global feature relationships. As the Transformer architecture has received widespread attention, Zhu et al. used its position encoding and global modeling capabilities to design an attention-guided non-uniform cropping strategy, which further improved the algorithm performance by removing a large number of uninformative patches in the reference aerial image and reallocating the saved computing power to useful information to improve image resolution. Summary of the invention

[0004] In order to at least partially solve the problem of how to establish feature associations of images from different perspectives based on the image domain gap between aerial images and ground images, the present invention provides a cross-perspective image geolocation method and system based on geometric relationship constraints. The present invention starts from the geometric mapping relationship between ground panoramic images and bird's-eye view images, converts the ground panoramic image to a bird's-eye view, and obtains an aerial image with obvious ground features. This conversion can fix the geometric similarity between the ground panoramic image and the satellite image, reducing the difficulty of algorithm network processing. The semantic segmentation network is used to extract key road information using the idea of ​​image segmentation to provide the visual content features required for cross-domain image matching; a global geometric layout feature extractor based on the Transformer architecture is used to extract the spatial configuration relationship between key features to constrain the drastic changes in background information. Finally, the designed feature self-interaction module is used to process the image background noise caused by the geometric transformation of the converted image, which overall enhances the model's ability to extract key features. The present invention establishes feature associations between aerial images and ground images based on geometric mapping relationships.

[0005] In order to achieve the above object, the technical solution of the present invention is:

[0006] The first aspect of the present invention proposes a cross-view image geolocation method based on geometric relationship constraints, comprising:

[0007] Step 1: Collect ground panoramic images and satellite images, and convert the ground panoramic images into bird's-eye view images according to the geometric mapping relationship to achieve domain transfer from ground panoramic images to bird's-eye view images, thus realizing cross-domain alignment of images.

[0008] Step 2: Input the bird's-eye view image and the satellite image into the feature extraction framework of CNN combined with Transformer to obtain the features of the bird's-eye view image and the satellite image, so as to facilitate feature extraction;

[0009] Step 3: Input the features of the bird's-eye view image and the features of the satellite image into the global feature aggregator, obtain the global descriptor of the bird's-eye view image and the global descriptor of the satellite image, calculate the loss between the two, and train the CNN combined with the Transformer feature extraction framework and the global feature aggregator by minimizing the overall loss;

[0010] Step 4: Collect satellite images of the target area, and input them into the trained CNN combined with the Transformer feature extraction framework and the global feature aggregator in turn to obtain the global descriptor of the satellite images of the target area, and construct it into a satellite image global feature descriptor database to facilitate matching and positioning with the ground panoramic image to be queried;

[0011] Step 5: Input the ground panoramic image to be queried into the trained CNN joint Transformer feature extraction framework and global feature aggregator in sequence. According to the obtained global descriptor of the bird's-eye view image to be queried, match the global descriptor of the bird's-eye view image to be queried with the global descriptor of the satellite image in the satellite image global feature descriptor database to complete the positioning of the ground panoramic image to be queried.

[0012] Furthermore, the geometric mapping relationship is expressed by the following formula:

[0013]

[0014] Among them, u p and v p are the pixel coordinates on the bird’s-eye view image plane, W b and H b are the width and length of the bird’s-eye view image, respectively, and W p and H p are the width and length of the ground panoramic image, f is the focal length of the bird's-eye view image, and arctan is the inverse tangent function.

[0015] Furthermore, the CNN combined with Transformer feature extraction framework includes a cross-domain visual content feature extractor, a global geometric layout feature extractor and a feature self-interaction module;

[0016] The cross-domain visual content feature extractor extracts features from the input ground panoramic image; the output of the cross-domain visual content feature extractor is respectively input into the global geometric layout feature extractor and the feature self-interaction module, and the global geometric layout feature extractor is used to process the output of the cross-domain visual content feature extractor; the feature self-interaction module is used to perform dot multiplication processing on the output of the cross-domain visual content feature extractor after being processed by the feature self-interaction module and the output of the global geometric layout feature extractor, so as to extract robust global geometric structure information, establish long-range dependency relationships between local features, and use the affinity relationship between local features to remove background noise caused by geometric transformation, thereby improving the robustness of key features.

[0017] Furthermore, the cross-domain visual content feature extractor includes a D-LinkNet network; the backbone network of the D-LinkNet network is a LinkNet network, and a hole convolution layer is provided between the encoder and the decoder of the D-LinkNet network;

[0018] The encoder of the LinkNet network is composed of a modified residual network, which includes the convolutional layers remaining after removing the 4th convolutional layer and the convolutional layers below the 4th convolutional layer in the standard residual network, so as to obtain intermediate local features;

[0019] The dilation rate in the dilated convolutional layer is expressed as follows:

[0020] K=k+(k-1)(r-1)

[0021] Among them, K is the actual convolution kernel size, k is the original convolution kernel size, and r is the expansion rate parameter.

[0022] Furthermore, the global geometric layout feature extractor includes a maximum pooling layer, a first embedding layer, a transformer encoder, a second embedding layer, and a position index perception unit arranged between the maximum pooling layer and the transformer encoder, which are connected in sequence. The position index perception unit is expressed by the following formula:

[0023]

[0024] Where E is the embedding vector, is the index-aware embedding vector, E pe is the learnable embedding vector, W LN is a learnable linear transformation, M idx is a parameter and HardTanh is the activation function.

[0025] Furthermore, the feature self-interaction module is used to perform dot multiplication processing on the output of the cross-domain visual content feature extractor after being processed by the feature self-interaction module and the output of the global geometric layout feature extractor, specifically including:

[0026] Flatten the output of the cross-domain visual content feature extractor;

[0027] Calculate the similarity matrix of the output of the flattened cross-domain visual content feature extractor, and sum the similarity matrix along the second coordinate axis to obtain an output vector;

[0028] The output vector is calculated using the TopK algorithm to obtain self-interaction features, which facilitates better distinction between the foreground and background information of the image;

[0029] The self-interaction features are dot-multiplied with the output of the global geometric layout feature extractor.

[0030] Furthermore, the similarity matrix is ​​expressed by the following formula:

[0031] S=h T ·h

[0032] Among them, S is the similarity matrix, S∈R N×N , N is the batch size, h is the flattened satellite image content feature or bird's-eye view image content feature, h T is the transpose of h.

[0033] Further, obtaining the global descriptor of the bird's-eye view image and the global descriptor of the satellite image and calculating the loss between the two specifically includes:

[0034]

[0035] Among them, L is the loss between the global descriptor of satellite image and the global descriptor of bird’s-eye view image, γ is a parameter, and d pos is the Euclidean distance between the matching images in the satellite image global descriptor and the bird’s-eye view image global descriptor, d neg is the Euclidean distance between the unmatched images in the satellite image global descriptor and the bird's-eye view image global descriptor.

[0036] The second aspect of the present invention proposes a cross-view image geolocation system based on geometric relationship constraints, comprising:

[0037] The conversion module is used to collect ground panoramic images and satellite images, and convert the ground panoramic images into bird's-eye view images according to the geometric mapping relationship, so as to realize the domain transfer from the ground panoramic images to the bird's-eye view images and realize the cross-domain alignment of images;

[0038] The feature extraction module is used to input the bird's-eye view image and the satellite image into the feature extraction framework of CNN combined with Transformer to obtain the features of the bird's-eye view image and the satellite image, so as to facilitate feature extraction;

[0039] A training module, used for inputting the features of the bird's-eye view image and the features of the satellite image into the global feature aggregator, obtaining a global descriptor of the bird's-eye view image and a global descriptor of the satellite image and calculating the loss between the two, and training the feature extraction framework of the CNN combined with the Transformer and the global feature aggregator by minimizing the overall loss;

[0040] The database module is used to collect satellite images of the target area, input the satellite images of the target area into the trained CNN combined with the Transformer feature extraction framework and the global feature aggregator in turn, obtain the global descriptor of the satellite images of the target area, and construct it into a satellite image global feature descriptor database to facilitate matching and positioning with the ground panoramic image to be queried;

[0041] The positioning module is used to input the ground panoramic image to be queried into the feature extraction framework and global feature aggregator of the trained CNN combined with Transformer in sequence, and match the global descriptor of the bird's-eye view image to be queried with the global descriptor of the satellite image in the satellite image global feature descriptor database according to the obtained global descriptor of the bird's-eye view image to be queried, so as to complete the positioning of the ground panoramic image to be queried.

[0042] Beneficial effects of the present invention:

[0043] (1) The present invention converts the surface panoramic image into a bird's-eye view image to obtain an aerial view image with obvious ground features. This conversion can fix the geometric similarity between the ground panoramic image and the satellite image, reduce the image domain gap between the aerial view image and the ground view image, and reduce the difficulty of subsequent processing.

[0044] (2) The present invention obtains a modified residual network by deleting the fourth and following convolutional layers in the standard residual network, thereby improving the ability to obtain intermediate local features, and setting a dilation rate parameter introduced by a dilated convolution to define the spacing between values ​​when the convolution kernel processes data, thereby expanding the receptive field level of the semantic segmentation network.

[0045] (3) The present invention realizes feature extraction of global spatial layout by introducing a position index perception unit into the global geometric layout feature extractor in the feature extraction framework. The feature self-interaction module is used to remove background noise, reduce the interference of background redundancy on feature extraction, and speed up the training speed of the feature extraction framework. Finally, the feature aggregator is used to improve the effectiveness and trainability of the global descriptor.

[0046] (4) The present invention trains the feature extraction framework by weighted soft margin triplet loss, which can make the matching image closer to the query image and the non-matching image farther away from the query image, thereby enhancing the accuracy of positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 One of the flowcharts of a cross-view image geolocation method based on geometric relationship constraints provided by an embodiment of the present invention.

[0048] Figure 2 The present invention provides a second flowchart of a method for cross-view image geolocation based on geometric relationship constraints in an embodiment of the present invention.

[0049] Figure 3 A schematic diagram of the imaging principle of a ground panoramic image in a cross-viewing image geolocation method based on geometric relationship constraints provided in an embodiment of the present invention.

[0050] Figure 4 A schematic diagram of the geometric mapping principle of a ground panoramic image in a cross-viewing image geolocation method based on geometric relationship constraints provided by an embodiment of the present invention.

[0051] Figure 5 A schematic diagram of the geometric mapping effect of a ground panoramic image of a cross-viewing image geolocation method based on geometric relationship constraints provided by an embodiment of the present invention.

[0052] Figure 6 A schematic diagram of a dilated convolution of a cross-view image geolocation method based on geometric relationship constraints provided in an embodiment of the present invention.

[0053] Figure 7 A schematic diagram of a global geometric layout feature extractor for a cross-view image geolocation method based on geometric relationship constraints provided in an embodiment of the present invention.

[0054] Figure 8 A schematic diagram of a feature self-interaction module of a cross-view image geolocation method based on geometric relationship constraints provided in an embodiment of the present invention.

[0055] Fig. 9 A schematic diagram of a feature aggregator of a cross-view image geolocation method based on geometric relationship constraints provided in an embodiment of the present invention.

[0056] Fig.10 An architectural diagram of a cross-viewing image geolocation system based on geometric relationship constraints provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Example 1

[0059] like Figure 1 As shown, the present invention provides a cross-viewing image geolocation method based on geometric relationship constraints, comprising:

[0060] S101: Collect ground panoramic images and satellite images, and convert the ground panoramic images into bird's-eye view images according to geometric mapping relationships.

[0061] S102: Input the bird's-eye view image and the satellite image into the feature extraction framework of CNN combined with Transformer to obtain features of the bird's-eye view image and features of the satellite image.

[0062] S103: Input the features of the bird's-eye view image and the features of the satellite image into the global feature aggregator, obtain the global descriptor of the bird's-eye view image and the global descriptor of the satellite image, calculate the loss between the two, and train the feature extraction framework of the CNN combined with the Transformer and the global feature aggregator by minimizing the overall loss.

[0063] S104: Collect satellite images of the target area, and input the satellite images of the target area into the trained CNN combined with the Transformer feature extraction framework and the global feature aggregator in sequence to obtain the global descriptor of the satellite images of the target area, and construct it into a satellite image global feature descriptor database.

[0064] S105: The ground panoramic image to be queried is sequentially input into the feature extraction framework and global feature aggregator of the trained CNN combined with the Transformer. According to the obtained global descriptor of the bird's-eye view image to be queried, the global descriptor of the bird's-eye view image to be queried is matched with the global descriptor of the satellite image in the global feature descriptor database of the satellite image, so as to complete the positioning of the ground panoramic image to be queried.

[0065] Specifically, the ground to be queried is located in the target area.

[0066] like Figure 2 As shown, the present invention first uses geometric mapping to convert the ground panoramic image into a bird's-eye view image to achieve cross-domain alignment of the image. The bird's-eye view image is then input into the feature extraction framework of the CNN combined with the Transformer, the key content features of the image are extracted by the cross-domain visual content feature extractor, and the robust global geometric structure information is extracted by the global geometric layout feature extractor, and the long-range dependency relationship between local features is established. Finally, the affinity relationship between local features is used by the feature self-interaction module to remove the background noise caused by geometric transformation, thereby improving the robustness of key features. The global feature aggregator is then used to aggregate the global descriptor of the bird's-eye view image and the global descriptor of the satellite image, and the weighted soft margin triplet loss is used for training to obtain the trained CNN combined with the Transformer feature extraction framework. The trained CNN combined with the Transformer feature extraction framework is used to complete the positioning of the ground panoramic image to be queried.

[0067] Example 2

[0068] Based on the above embodiments, the embodiments of the present invention provide a specific process for converting a ground panoramic image into a bird's-eye view image according to a geometric mapping relationship, which specifically includes:

[0069] There is a fixed geometric projection relationship between the satellite image and the ground panoramic image taken at its coordinate center. This geometric projection relationship can be used to realize the conversion between the ground panoramic image and the satellite image. However, this conversion requires the participation of photography parameters such as the shooting height of the ground camera, which is difficult to achieve in real application scenarios. Therefore, the present invention explores a geometric mapping method based on the imaging principle of ground panoramic images to realize the domain transfer of ground panoramic images to bird's-eye view images.

[0070] The imaging method of ground panoramic camera is as follows Figure 3 As shown, it captures a full 360° view on a spherical projection plane and displays it through equirectangular projection, that is, a cylinder is used to represent the image surface, but the image pixels are parameterized in a spherical coordinate system. Let P = (x1, y1, z1) be a point in the ground camera coordinate system, P′ = (x2, y2) represent the projection point in the normalized equirectangular coordinate system of the panorama, (ψ, θ) be the spherical coordinates, which can be obtained from (x1, y1, z1) using inverse trigonometric functions, as shown in formulas (1) and (2). The conversion formulas between spherical coordinates (ψ, θ) and normalized equirectangular coordinates (x2, y2) of the panoramic image are shown in (3) and (4).

[0071]

[0072]

[0073] Among them, ψ and θ are points in spherical coordinates, atan2 is the four-quadrant inverse tangent function, x1, y1 and z1 are points in the ground camera coordinate system, sign is the sign function, and x2 and y2 are normalized rectangular coordinate points of the panoramic image.

[0074] In order to realize the mapping from ground panoramic image to bird's-eye view image, a tangent plane is placed at the south pole of the spherical imaging plane as the new imaging plane, such as Figure 4 As shown. Assume the size of the panoramic image is H p ×W p , the size of the bird's-eye view image after geometric mapping conversion is H b ×W b Connect the optical center of the camera to each pixel point on the bird's-eye view imaging plane, and determine the corresponding pixel position in the panoramic image through the intersection of the connecting line and the spherical imaging plane. Suppose the pixel coordinates on the bird's-eye view imaging plane are (u b ,v b ), we can directly get the corresponding camera coordinates (x1, y1, z1). Set the mapping perspective parameter fov = 85° to determine the field of view of the bird's-eye view image. The focal length of the bird's-eye view image during imaging is:

[0075]

[0076] Where f is the focal length of the bird’s-eye view image during imaging, W b is the width of the bird's-eye view image, tan is the tangent function, and fov is the mapping viewing angle parameter.

[0077] Therefore, the geometric mapping relationship between the ground panoramic image and the bird's-eye view image can be expressed as:

[0078]

[0079] Among them, u p and v p are the pixel coordinates on the bird’s-eye view image plane, W b and H b are the width and length of the bird’s-eye view image, respectively, and W p and H p are the width and length of the ground panoramic image, f is the focal length of the bird's-eye view image, and arctan is the inverse tangent function.

[0080] Using the principle of geometric mapping to process ground panoramic images can effectively align the feature domains of the two images, thereby constraining the image domain gap caused by the change in viewing angle, making the corresponding feature maps closer in the two feature spaces. The effect of geometric transformation is as follows: Figure 5 shown.

[0081] From the conversion effect, facilities such as roads located on the ground plane have been well restored and have a great appearance similarity with real satellite images. This makes it easier for the feature extraction network to extract more discriminative features when performing cross-view image matching and positioning tasks, and makes the image matching process more interpretable.

[0082] Example 3

[0083] Based on the above embodiments, the embodiment of the present invention provides a specific structure of a feature extraction framework of CNN combined with Transformer, which specifically includes:

[0084] The feature extraction framework of CNN combined with Transformer includes a cross-domain visual content feature extractor, a global geometric layout feature extractor and a feature self-interaction module. The output of the cross-domain visual content feature extractor is input into the global geometric layout feature extractor, and the output of the cross-domain visual content feature extractor is processed by the feature self-interaction module and then dot-multiplied with the output of the global geometric layout feature extractor.

[0085] The basic operation of the ground panoramic image geometric projection is to map the pixels of the ground image to the ground plane. When the side view pixels above the ground plane are mapped, they will be superimposed together to produce a large distortion, such as Figure 5As shown. Due to the abstract problems such as missing details and spatial dislocation in the converted images, there is still a big gap between the two. Therefore, the ground panoramic image and satellite image after geometric mapping transformation only achieve basic geometric alignment. Information such as roads flush with the ground plane will be retained to the greatest extent after mapping, which is also the most critical feature for matching with satellite images. Therefore, selecting a feature extraction network that can separate road information is the key to solving the cross-domain image matching problem. In order to meet this challenge, the natural characteristics of the road must be taken into account to extract the road features as completely as possible from the complex background of the image. First of all, the roads in satellite images are often slender, complex, and cover a small part of the entire image. Secondly, the roads have natural connectivity and long spans. Based on these two characteristics, it is determined that the feature extraction network must have a large receptive field covering the entire image and be able to retain feature details. Therefore, the cross-domain visual content feature extractor of the present invention chooses to use the D-LinkNet network of semantic segmentation.

[0086] The D-LinkNet network uses LinkNet with a pre-trained encoder as the backbone network, and uses additional hole convolution layers to connect the central part of the network encoding-decoding structure. LinkNet is an efficient semantic segmentation neural network that uses skip connections, residual blocks, and encoding-decoding structures. The LinNet here uses the residual network (ResNet50) operator as the encoder. However, because ResNet50 has deeper convolutional layers, resulting in a higher semantic level and a larger receptive field, the network will lack the ability to grasp the details of the overall features. Therefore, the fourth convolutional layer (conv4 layer) and the following layers of ResNet50 are removed to obtain intermediate local features. At the same time, in order to make the network have the receptive field level that only deep features have, hole convolutions are added after the encoding network. The principle of hole convolution is as follows Figure 6 As shown in the figure. Because a considerable amount of information will be lost when the receptive field is expanded during the ordinary convolution pooling process, this makes the extraction of street scene feature extraction branches, which are originally affected by image noise and distortion, even more difficult. Compared with ordinary convolution, the dilated convolution introduces a new parameter called "expansion rate". This parameter defines the spacing between the values ​​when the convolution kernel processes data, and expands the convolution kernel size without adding additional parameters. The formula is shown in (8):

[0087] K=k+(k+1)(r-1) (8)

[0088] Among them, K is the actual convolution kernel size, k is the original convolution kernel size, and r is the expansion rate parameter.

[0089] It can be calculated that when the original convolution kernel size is 3 and the dilation rate is 2, the actual convolution kernel size of the dilated convolution is 5. Therefore, the dilated convolution can expand the receptive field without losing image resolution and obtain multi-scale contextual information.

[0090] Because cross-view images lack sufficient visual similarity features to support matching and positioning, it is necessary to mine other stable information between images to provide gain for cross-view image matching. Among them, the global spatial layout not only reflects the position between visual features, but also can reflect the global context information of the image to the greatest extent. This spatial layout is the most stable relationship when the image perspective changes, which is called geometric constraint under perspective change. Especially for the matching between bird's-eye view images and real satellite images after perspective conversion, a large amount of geometric distortion makes the street view images after perspective conversion lose most of the visual features and image details, and the spatial layout between roads is the best supplement. Therefore, in order to achieve the purpose of extracting global spatial layout features, the feature extraction framework of CNN combined with Transformer designs a global geometric layout feature extractor based on the Transformer architecture. The global geometric layout feature extractor can take advantage of the advantages of Transformer, that is, establish the dependency relationship of global features, thereby generating a geometric layout descriptor between local features extracted by the CNN network, and generating supplementary context information for the final global descriptor.

[0091] To achieve this goal, the global geometric layout feature extractor structure is established as follows Figure 7 As shown in Figure 1, the global geometric layout feature extractor consists of a channel maximum pooling layer, a Transformer encoder, and two embedding layers. The maximum pooling layer produces a saliency map image M∈R H×W , where H is the height and W is the width. M is then processed by the first embedding layer and projected into K embedding vectors E = [e1, e2, ..., e k ], e k is the kth embedding vector, each of which has a dimension of Embedding (PE) in standard learnable positions pe Above, an additional index-aware position embedding is introduced and added to E, as shown in formula (8).

[0092]

[0093] Where E is the embedding vector, is the index-aware embedding vector, E pe is the learnable embedding vector, W LN is a learnable linear transformation, The main function is to idxMapped to K different subspaces, M idx As parameters, argmax is the maximum independent variable point set, c is the number of channels, and HardTanh is the activation function.

[0094] The Transformer encoder is mainly used to extract The correlation between the features is then projected through the second embedding layer to generate a set of K geometric layout descriptors. Finally, after the activation function and multiplication with the original features, a feature map with half the size is obtained.

[0095] The process of geometric mapping of ground panoramic images is a pure geometric transformation, which will cause obvious distortion in the transformation result, which is called background noise. Even if the present invention uses the strategy of image segmentation to extract key road information, a large amount of background redundancy will still cause great interference to feature extraction and affect the network training speed. In the street view image feature extraction branch of the present invention, a feature self-interaction module is added to better distinguish the foreground information and background information of the image.

[0096] By observing the transformed street view image, we can see that the redundant background information is distributed in blocks, and the changes between blocks are much smaller than the foreground road information, which means that the background has more similar features than the foreground road. Based on this observation, a heuristic module called the self-interaction module is proposed. The schematic diagram is shown in the figure. Figure 8 This module eliminates redundant background features by calculating the similarity between the same transformed street view images. Specifically, given the feature map g∈R output by the local feature extractor C×H×W , the self-interaction module first flattens the satellite image content features and the bird's-eye view image content features: h = flatten(g), h∈R C×N Where N = H × W, then we can calculate the similarity matrix, which is expressed as follows:

[0097] S=h T ·h (10)

[0098] Among them, S is the similarity matrix, S∈R N×N , N is the batch size, h is the flattened satellite image content feature or bird's-eye view image content feature, h T is the transpose of h.

[0099] In the similarity matrix, the row vector represents the similarity between a local feature and all local features. According to the above assumption, background features have more similar local features, which means that the number of large values ​​in the row vector of background features should be more than that of foreground features. Therefore, the similarity matrix is ​​summed along the second coordinate axis to obtain the satellite image vector and the bird's-eye view image vector, in which the larger value may represent the background feature in the local feature. Finally, the TopK algorithm is applied to the satellite image vector and the bird's-eye view image vector to filter out the satellite image self-interaction features and the bird's-eye view image self-interaction features. in, is the satellite image self-interaction feature or the bird's-eye view image self-interaction feature, o is the ratio of the object feature, and in the present invention, o=0.5.

[0100] Example 3

[0101] Based on the above embodiment, the embodiment of the present invention provides a specific structure of a global feature aggregator, which specifically includes:

[0102] In order to cope with the drastic difference in perspective, global feature descriptors are often used in cross-perspective image matching and positioning tasks based on image retrieval, so a suitable feature aggregation strategy is particularly important. The present invention adopts MixVPR, the most advanced feature aggregation strategy currently used in geolocation tasks. MixVPR uses the feature map extracted by the pre-trained backbone network as input, and then uses multiple feature mixers with the same structure and composed entirely of multi-layer perceptrons to iteratively incorporate global relationships into each individual feature map, thereby eliminating the need for local or pyramid aggregation in NetVLAD, greatly improving the effectiveness and trainability of the global descriptor. Its principle structure diagram is shown in the figure. Fig. 9 shown.

[0103] For a given input F∈R c×n , c represents the number of feature channels, and n represents the number of input features. The input features are first flattened and then processed using the feature mixer (xL). Specifically, the flattened input is first input into the first feature mixer block to produce an output of the same shape, which is then sent to the second feature mixer block, and so on, until L consecutive blocks are reached. The formula is as follows:

[0104] Z=FM L (FM L-1 (...FM1(F))) (11)

[0105] Among them, Z is the mixed feature, F is the input feature, M L A feature mixer.

[0106] The dimension of Z is the same as the input features. To further reduce its dimensionality, two fully connected layers are added after the feature mixer to reduce its depth (channel) and row dimensions respectively. This operation can be regarded as a weighted pooling operation to control the size of the final global descriptor. First, Z is transformed from R to R using depth projection. c×n Mapping to R d×n , as shown below:

[0107] Z′=W d (Transpose(Z)) (12)

[0108] Among them, Z′ is the input feature of the first dimension reduction, W d is the weight of the fully connected layer, and Transpose is the mapping function.

[0109] Then apply a row-by-row projection, transforming the output Z' from R d×n Mapping to R d×r , as shown in formula (13):

[0110] O=W r (Transpose(Z′)) (13)

[0111] Among them, O is the input feature of the second dimensionality reduction, the dimension of O is d×r, and W r is the weight of the fully connected layer.

[0112] Finally, the input features of the second dimensionality reduction are flattened and Euclidean normalized (L2 normalization), and the global descriptors of satellite images and bird's-eye view images are output.

[0113] Example 4

[0114] Based on the above embodiment, the embodiment of the present invention provides a specific process of training a feature extraction framework of CNN combined with Transformer, which specifically includes:

[0115] The weighted soft-margin triplet loss is usually regarded as the most suitable objective function for geolocation tasks based on deep metric learning. The main purpose of the triplet loss is to make the matching image closer to the query image and the non-matching image farther away from the query image by learning the distance metric between feature descriptors. The formula is shown in (14):

[0116]

[0117] Among them, L is the loss between the global descriptor of satellite image and the global descriptor of bird’s-eye view image, γ is a parameter, and d pos is the Euclidean distance between the matching images in the satellite image global descriptor and the bird’s-eye view image global descriptor, d negis the Euclidean distance between the unmatched images in the satellite image global descriptor and the bird's-eye view image global descriptor.

[0118] The loss between the global descriptor of the satellite image and the global descriptor of the bird's-eye view image is calculated by weighted soft margin triplet loss, and then the feature extraction framework of CNN combined with Transformer and the global feature aggregator are optimized.

[0119] Example 5

[0120] Corresponding to the above method, such as Fig.10 As shown, an embodiment of the present invention provides a cross-viewing image geolocation system based on geometric relationship constraints, including:

[0121] A conversion module is used to collect ground panoramic images and satellite images, and convert the ground panoramic images into bird's-eye view images according to geometric mapping relationships;

[0122] The feature extraction module is used to input the bird's-eye view image and the satellite image into the feature extraction framework of CNN combined with Transformer to obtain the features of the bird's-eye view image and the features of the satellite image;

[0123] A training module, for inputting the features of the bird's-eye view image and the features of the satellite image into a global feature aggregator, obtaining a global descriptor of the bird's-eye view image and a global descriptor of the satellite image and calculating the loss between the two, and training the feature extraction framework of the CNN combined with the Transformer and the global feature aggregator by minimizing the overall loss;

[0124] The database module is used to collect satellite images of the target area, input the satellite images of the target area into the trained CNN combined with the Transformer feature extraction framework and the global feature aggregator in turn, obtain the global descriptor of the satellite images of the target area, and construct it into a satellite image global feature descriptor database;

[0125] The positioning module is used to input the ground panoramic image to be queried into the feature extraction framework and global feature aggregator of the trained CNN combined with Transformer in sequence, and match the global descriptor of the bird's-eye view image to be queried with the global descriptor of the satellite image in the satellite image global feature descriptor database according to the obtained global descriptor of the bird's-eye view image to be queried, so as to complete the positioning of the ground panoramic image to be queried.

[0126] It should be noted that the cross-view image geolocation system based on geometric relationship constraints provided in an embodiment of the present invention is for realizing a cross-view image geolocation method based on geometric relationship constraints in the above-mentioned embodiment. Its specific functions can be referred to the above-mentioned method embodiments and will not be repeated here.

[0127] In summary, the present invention converts the panoramic image into a bird's-eye view image to obtain an aerial image with obvious ground features. This conversion can fix the geometric similarity between the ground panoramic image and the satellite image, reduce the image domain gap between the aerial image and the ground image, and reduce the difficulty of subsequent processing. The present invention obtains a modified residual network by deleting the fourth and following convolutional layers in the standard residual network, improves the ability to obtain intermediate local features, and sets a dilation rate parameter introduced by the dilation convolution to define the spacing of each value when the convolution kernel processes data, thereby expanding the receptive field level of the semantic segmentation network. The present invention realizes feature extraction of the global spatial layout by introducing a position index perception unit in the global geometric layout feature extractor in the feature extraction framework. The feature self-interaction module is used to remove background noise, reduce the interference of background redundancy on feature extraction, and speed up the training speed of the feature extraction framework. Finally, the feature aggregator improves the effectiveness and trainability of the global descriptor. The present invention trains the feature extraction framework by weighted soft margin triplet loss, which can make the matching image closer to the query image, and the unmatched image is far away from the query image, thereby enhancing the accuracy of positioning.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-view image geolocation method based on geometric relationship constraints, characterized in that: include: Step 1: Collect ground panoramic images and satellite images, and convert the ground panoramic images into bird's-eye view images based on geometric mapping relationships; Step 2: Input the bird's-eye view image and the satellite image into the feature extraction framework of CNN combined with Transformer to obtain the features of the bird's-eye view image and the features of the satellite image; Step 3: Input the features of the bird's-eye view image and the features of the satellite image into the global feature aggregator, obtain the global descriptor of the bird's-eye view image and the global descriptor of the satellite image, calculate the loss between the two, and train the CNN combined with the Transformer feature extraction framework and the global feature aggregator by minimizing the overall loss; Step 4: Collect satellite images of the target area, and input the satellite images of the target area into the trained CNN combined with the Transformer feature extraction framework and the global feature aggregator in turn to obtain the global descriptor of the satellite images of the target area, and construct it into a satellite image global feature descriptor database; Step 5: Input the ground panoramic image to be queried into the trained CNN joint Transformer feature extraction framework and global feature aggregator in sequence. According to the obtained global descriptor of the bird's-eye view image to be queried, match the global descriptor of the bird's-eye view image to be queried with the global descriptor of the satellite image in the satellite image global feature descriptor database to complete the positioning of the ground panoramic image to be queried.

2. The method for cross-view image geolocation based on geometric relationship constraints according to claim 1, characterized in that: The geometric mapping relationship is expressed by the following formula: Among them, u p and v p are the pixel coordinates on the bird’s-eye view image plane, W b and H b are the width and length of the bird’s-eye view image, respectively, and W p and H p are the width and length of the ground panoramic image, f is the focal length of the bird's-eye view image, and arctan is the inverse tangent function.

3. The method for cross-view image geolocation based on geometric relationship constraints according to claim 1, characterized in that: The CNN combined with Transformer feature extraction framework includes a cross-domain visual content feature extractor, a global geometric layout feature extractor and a feature self-interaction module; The cross-domain visual content feature extractor extracts features from the input ground panoramic image; the output of the cross-domain visual content feature extractor is respectively input into the global geometric layout feature extractor and the feature self-interaction module, and the global geometric layout feature extractor is used to process the output of the cross-domain visual content feature extractor; the feature self-interaction module is used to perform dot multiplication processing on the output of the cross-domain visual content feature extractor after being processed by the feature self-interaction module and the output of the global geometric layout feature extractor.

4. The method for cross-view image geolocation based on geometric relationship constraints according to claim 3, characterized in that: The cross-domain visual content feature extractor includes a D-LinkNet network; the backbone network of the D-LinkNet network is a LinkNet network, and a hole convolution layer is arranged between the encoder and the decoder of the D-LinkNet network; The encoder of the LinkNet network is composed of a modified residual network, wherein the modified residual network includes the convolutional layers remaining after removing the 4th convolutional layer and the convolutional layers below the 4th convolutional layer in the standard residual network; The dilation rate in the dilated convolutional layer is expressed as follows: K=k+(k-1)(r-1) Among them, K is the actual convolution kernel size, k is the original convolution kernel size, and r is the expansion rate parameter.

5. The method for cross-view image geolocation based on geometric relationship constraints according to claim 3, characterized in that: The global geometric layout feature extractor includes a maximum pooling layer, a first embedding layer, a transformer encoder, a second embedding layer, and a position index perception unit arranged between the maximum pooling layer and the transformer encoder, which are connected in sequence. The position index perception unit is expressed by the following formula: Where E is the embedding vector, is the index-aware embedding vector, E pe is the learnable embedding vector, W LN is a learnable linear transformation, M idx is a parameter and HardTanh is the activation function.

6. The method for cross-view image geolocation based on geometric relationship constraints according to claim 3, characterized in that: The feature self-interaction module is used to perform dot multiplication processing on the output of the cross-domain visual content feature extractor after being processed by the feature self-interaction module and the output of the global geometric layout feature extractor, specifically including: Flatten the output of the cross-domain visual content feature extractor; Calculate the similarity matrix of the output of the flattened cross-domain visual content feature extractor, and sum the similarity matrix along the second coordinate axis to obtain an output vector; The output vector is calculated using the TopK algorithm to obtain the self-interaction feature; The self-interaction features are dot-multiplied with the output of the global geometric layout feature extractor.

7. The method for cross-view image geolocation based on geometric relationship constraints according to claim 6, characterized in that: The similarity matrix is ​​expressed by the following formula: S=h T ·h Among them, S is the similarity matrix, S∈R N×N , N is the batch size, h is the flattened satellite image content feature or bird's-eye view image content feature, h T is the transpose of h.

8. The method for cross-view image geolocation based on geometric relationship constraints according to claim 1, characterized in that: The obtaining of the global descriptor of the bird's-eye view image and the global descriptor of the satellite image and calculating the loss between the two specifically includes: Among them, L is the loss between the global descriptor of satellite image and the global descriptor of bird’s-eye view image, γ is a parameter, and d pos is the Euclidean distance between the matching images in the satellite image global descriptor and the bird’s-eye view image global descriptor, d neg is the Euclidean distance between the unmatched images in the satellite image global descriptor and the bird's-eye view image global descriptor.

9. A cross-view image geolocation system based on geometric relationship constraints, characterized in that: include: A conversion module is used to collect ground panoramic images and satellite images, and convert the ground panoramic images into bird's-eye view images according to geometric mapping relationships; The feature extraction module is used to input the bird's-eye view image and the satellite image into the feature extraction framework of CNN combined with Transformer to obtain the features of the bird's-eye view image and the features of the satellite image; A training module, for inputting the features of the bird's-eye view image and the features of the satellite image into a global feature aggregator, obtaining a global descriptor of the bird's-eye view image and a global descriptor of the satellite image and calculating the loss between the two, and training the feature extraction framework of the CNN combined with the Transformer and the global feature aggregator by minimizing the overall loss; The database module is used to collect satellite images of the target area, input the satellite images of the target area into the trained CNN combined with the Transformer feature extraction framework and the global feature aggregator in turn, obtain the global descriptor of the satellite images of the target area, and construct it into a satellite image global feature descriptor database; The positioning module is used to input the ground panoramic image to be queried into the feature extraction framework and global feature aggregator of the trained CNN combined with Transformer in sequence, and match the global descriptor of the bird's-eye view image to be queried with the global descriptor of the satellite image in the satellite image global feature descriptor database according to the obtained global descriptor of the bird's-eye view image to be queried, so as to complete the positioning of the ground panoramic image to be queried.

Images