Spatial perception geolocation method and system for panoramic images

Abstract

The present invention provides a spatial perception geolocation method and system for panoramic images, which are characterized by: establishing a data set of panoramic images and optical remote sensing images storing geographic coordinates, performing polar coordinate conversion on the optical remote sensing images to align them with the panoramic images; constructing a spatial perception geolocation neural network composed of a backbone network, a spatial perception module and a sky view factor automatic calculation branch, and training the neural network based on a binomial loss function that balances positive and negative samples; applying an attention mechanism to image feature descriptors generated during training to obtain image feature descriptors that suppress deformation, and using a spatial perception module to aggregate and embed the image feature descriptors to generate a final image feature descriptor that suppresses deformation; calculating the sky view factor of the panoramic image by the sky view factor automatic calculation branch; embedding the sky view factor into the final image feature descriptor that suppresses deformation to generate a new image feature descriptor; and performing geolocation based on the new image feature descriptor.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a spatial perception geolocation method and system for panoramic images. Background Art

[0002] The image geolocation task refers to determining the geographical location of the image content only through the visual information given by the image. This task has a wide range of application scenarios, such as unmanned driving, VR / AR, and auxiliary robots. The traditional image geolocation algorithm refers to comparing and matching the captured panoramic image with the panoramic image with geographic location information in the database, but this method has certain defects: on the one hand, due to the large number of replica buildings around the world, landmarks in the image may still be impossible to locate; on the other hand, the panoramic images with geographic location information in the database are mostly concentrated in cities and densely populated areas, and it is difficult to obtain accurate geographic location tags in many remote rural areas and sparsely populated areas. Due to the limitation of the viewing angle range of the reference image on the ground, this method is difficult to achieve full coverage of the ground, and has the disadvantages of inaccurate positioning and a small number of locatable points.

[0003] Optical remote sensing images can provide global geographic coordinates with wide coverage and high accuracy, so matching panoramic images with optical remote sensing images in the database becomes an attractive alternative. However, there is a huge difference in perspective between optical remote sensing images and panoramic images, making this cross-perspective image geolocation task very challenging. Therefore, it is urgent to design a technical solution that can overcome the above defects. Summary of the invention

[0004] In order to solve the above problems, a method for geo-locating a panoramic image with embedded scene content is provided. The present invention adopts the following technical solutions:

[0005] The present invention provides a spatial perception geolocation method for a panoramic image, which is characterized by comprising the following steps: step S1, establishing a data set including a panoramic image and an optical remote sensing image storing geographic coordinates, performing polar coordinate conversion on the optical remote sensing image, so that the optical remote sensing image and the panoramic image are aligned accordingly; step S2, constructing a spatial perception geolocation neural network, the spatial perception geolocation neural network having a backbone network, a spatial perception module and a sky view factor automatic calculation branch; step S3, using the data set to train the spatial perception geolocation neural network based on a predetermined loss function, applying an attention mechanism to image feature descriptors generated during the training process, and obtaining an image feature descriptor that suppresses deformation; step S4, based on the spatial perception module, performing aggregation embedding on the image feature descriptors that suppress deformation to generate a final image feature descriptor that suppresses deformation; step S5, calculating the sky view factor of the panoramic image based on the sky view factor automatic calculation branch; step S6, embedding the sky view factor into the final image feature descriptor that suppresses deformation to generate a new image feature descriptor; step S7, the spatial perception geolocation neural network performs geolocation based on the new image feature descriptor to obtain an optical remote sensing image and geographic coordinates corresponding to the panoramic image.

[0006] The spatial perception geolocation method of the panoramic image provided by the present invention may also have the following technical features: in step S1, a ×S a Optical remote sensing images The polar coordinate transformation is performed to obtain the size of H after polar coordinate transformation. g ×W g The target image

[0007]

[0008]

[0009] The spatially-aware geo-positioning method for panoramic images provided by the present invention may also have such a technical feature, wherein step S3 comprises the following sub-steps: step S3-1, selecting the most unique object features along the feature channel c using a maximum pooling operator; step S3-2, using a spatially-aware importance generator to generate a position embedding map P∈R H×W ; Step S3-3, in the spatially aware importance generator, two fully connected layers are used to select features f in the prominent layer c ∈R H×W , and calculate the Frobenius inner product of the spatial combination and feature response, and encode and generate an image feature descriptor that suppresses deformation:

[0010] K={k c},c=1,2,……,C

[0011] k c = <f c ,P> F

[0012] In the formula, f c ∈R H×W represents the input feature map of the spatially aware position embedding module in the cth channel, <.,.> represents the Frobenius inner product of the two inputs, and k c is the embedded feature activation of the c-th channel.

[0013] The spatial perception geolocation method for panoramic images provided by the present invention may also have such a technical feature, wherein, in step S4, the spatial perception module has a plurality of spatial perception position embedding blocks with the same architecture but different weights, and a plurality of embedding graphs P∈R are generated based on the plurality of spatial perception position embedding blocks. H×W , and then encode the input features according to the different generated masks, and connect the embedded features together as the final deformation-suppressed image feature descriptor.

[0014] The spatial perception geolocation method of the panoramic image provided by the present invention may also have such a technical feature, wherein step S5 includes the following sub-steps: step S5-1, using a pyramid scene parsing network to segment the sky, trees, and buildings in the panoramic image, and marking them with white, green, and orange respectively; step S5-2, for the segmented panoramic image (x p ,y p ) is used for equiangular fisheye projection, which is then transformed into coordinates (x f ,y f ) and the fisheye image of the sky hemisphere with polar coordinates (r,θ):

[0015]

[0016]

[0017] Step S5-3, dividing the sky hemispherical fisheye image into n annular rings, and adding the contribution of each annular ring to obtain the sky viewing factor of the panoramic image:

[0018]

[0019] Where SVF is the sky view factor, is the ratio between the number of pixels in the sky area and the total number of pixels in ring i.

[0020] The spatially aware geo-positioning method for panoramic images provided by the present invention may also have the following technical features: in step S7, the new image feature descriptor is:

[0021] K′=SVF{k c}

[0022] Where SVF is the sky view factor, k c is the embedded feature activation of the c-th channel.

[0023] The spatially aware geo-positioning method for panoramic images provided by the present invention may also have such a technical feature, wherein the predetermined loss function is a binomial loss function that balances positive and negative samples:

[0024]

[0025] In the formula, m represents pushing the negative samples away by a margin of m. and represents the cosine similarity between the i-th anchor point and its corresponding positive and negative samples, N p and N n Represent the number of positive and negative sample pairs, α p and α n Represents positive and negative samples being pulled in and pushed away with different weights.

[0026] The present invention also provides a spatial perception geolocation system for panoramic images, characterized in that it includes: a geometric domain conversion module, a backbone network module, a spatial perception feature aggregation module, a sky view factor module and an input display module; wherein the geometric domain conversion module is used to perform polar coordinate transformation on an optical remote sensing image data set storing geographic coordinates so as to align the optical remote sensing image and the panoramic image accordingly, and the backbone network module is used to train a twin deep neural network based on a binomial loss function of balanced positive and negative samples using the panoramic image data set and the optical remote sensing image data set after polar coordinate transformation to generate a spatial perception geolocation neural network, and obtain information related to the spatial perception geolocation neural network based on the spatial perception geolocation neural network. The optical remote sensing image and geographic coordinates corresponding to the panoramic image, the spatial perception feature aggregation module is used to encode the important features extracted by the spatial perception geographic positioning network and the relative positions of the important features to generate an image feature descriptor that suppresses deformation, and aggregate and embed the image feature descriptor that suppresses deformation, and connect the embedded features together as the final image feature descriptor that suppresses deformation. The sky view factor module has a panoramic image scene analysis unit, an equiangular fisheye projection unit and a sky view factor numerical calculation unit. The input display module has a screen storage unit, and the screen storage unit has a picture upload screen, a query running window screen, a result query screen and a historical query result screen.

[0027] The present invention also provides an electronic device, characterized in that it includes: at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned spatial-aware geolocation method of panoramic images.

[0028] The present invention also provides a computer-readable storage medium for storing a computer program, characterized in that the computer program is configured to implement the steps of the spatially-aware geo-positioning method of a panoramic image when executed.

[0029] Function and Effect of the Invention

[0030] According to the spatial perception geolocation method of panoramic images of the present invention, the optical remote sensing images in the data set are converted into polar coordinates, thus bridging the geometric domain difference between them and the panoramic images. The constructed spatial perception geolocation neural network is an image geolocation neural network that is embedded with spatial perception feature aggregation of scene content, in which each branch of the twin deep neural network can learn the deep features matched between the two diagrams of the panoramic image and the optical remote sensing image without sharing weights; the spatial perception module can encode the extracted important features and the relative positions between the features, thereby generating an image feature descriptor that suppresses deformation and avoids the deformation problem caused by polar coordinate conversion. In addition, since the sky view factor calculation branch in the spatial perception geolocation neural network can parse the scene content of the panoramic image, it effectively pays attention to the spatial position information of the panoramic image, so that the spatial perception geolocation method of the panoramic image of the present invention has better positioning accuracy. At the same time, due to the low computational complexity of the spatial perception module, the performance of panoramic image geolocation is also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of a spatially aware geo-positioning method for a panoramic image in an embodiment of the present invention;

[0032] Figure 2 is a schematic diagram of geo-positioning using a spatially aware geo-positioning method based on panoramic images according to an embodiment of the present invention;

[0033] Figure 3 is a comparison chart of experimental evaluation results of the spatial perception geolocation method of panoramic images and the traditional geolocation method in an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of the results of verifying the effectiveness of polar coordinate transformation for panoramic image geolocation in an embodiment of the present invention;

[0035] Figure 5is a schematic diagram of the results of verifying the effectiveness of multiple spatially aware position embedding modules for panoramic image geolocation in an embodiment of the present invention;

[0036] Figure 6 is a schematic diagram of the results of verifying the effectiveness of the binomial loss function for balancing positive and negative samples for panoramic image geolocation in an embodiment of the present invention;

[0037] Figure 7 A schematic diagram of the results of verifying the effectiveness of the sky view factor automatic calculation branch for panoramic image geolocation in an embodiment of the present invention;

[0038] Figure 8 is a schematic diagram of the structure of a spatial perception geo-positioning system for panoramic images according to an embodiment of the present invention;

[0039] Fig. 9 is a schematic diagram of a picture uploading screen of a spatial perception geographic positioning system for panoramic images in an embodiment of the present invention;

[0040] Fig.10 A schematic diagram of a query operation window screen of a spatial perception geographic positioning system for panoramic images according to an embodiment of the present invention;

[0041] Fig.11 is a schematic diagram of a result query screen of a spatially aware geographic positioning system for panoramic images in an embodiment of the present invention;

[0042] Fig.12 It is a schematic diagram of a historical query result screen of a spatial perception geographic positioning system for panoramic images in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the spatial perception geo-positioning method and system of panoramic images of the present invention are specifically described below in combination with embodiments and drawings.

[0044] <Example>

[0045] Figure 1 is a flow chart of a spatially aware geo-positioning method for a panoramic image in an embodiment of the present invention, Figure 2 It is a schematic diagram of geo-positioning using a spatially aware geo-positioning method based on panoramic images according to an embodiment of the present invention.

[0046] like Figure 1 and Figure 2 As shown, the spatially aware geolocation method of panoramic images comprises the following steps:

[0047] Step S1, establishing an optical remote sensing image data set including a panoramic image and stored geographic coordinates, performing polar coordinate conversion on the optical remote sensing image, so that the spatial layout of the optical remote sensing image and the panoramic image are aligned accordingly.

[0048] In this embodiment, the panoramic images in the panoramic image dataset use equirectangular rectangular projection to project 360-degree rays onto the image plane and are orthogonal to the satellite view image, so that the vertical lines in the panoramic images correspond to the radial lines in the optical remote sensing images, and the horizontal lines in the panoramic images approximately correspond to the circle lines in the optical remote sensing images.

[0049] Therefore, in the embodiment, the size of S a ×S a Optical remote sensing images The polar coordinate transformation is performed to obtain the size of H after polar coordinate transformation. g ×W g The target image The conversion formula is as follows:

[0050]

[0051]

[0052] Step S2, constructing a spatially aware geolocation neural network.

[0053] In this embodiment, the spatial perception geolocation neural network is composed of a backbone network, a spatial perception module, and a sky view factor automatic calculation branch, and is an image geolocation neural network that aggregates spatial perception features embedded in scene content.

[0054] Among them, the backbone network is a twin deep neural network. Different from the twin deep neural network previously used for traditional image geolocation, each branch of this twin deep neural network learns deep features suitable for matching between two views and the two branches do not share weights. Because the domains of panoramic images and optical remote sensing images are different, it is beneficial to give the weights of each branch more degrees of freedom.

[0055] As a plug-in that does not change the original network structure, the spatial perception module only constrains the learning of hidden layer features through back propagation. The spatial perception module consists of a spatial perception importance generator and multiple spatial perception position embedding modules with the same architecture but different weights.

[0056] Step S3, using the image data set to train the spatial perception geolocation neural network based on a predetermined loss function, applying an attention mechanism to the image feature descriptors of the panoramic images and optical remote sensing images generated during the training process, encoding the relative positions and important features of the object features extracted by the twin deep neural network, and obtaining an image feature descriptor that suppresses deformation.

[0057] The conventional image geolocation loss function is formulated as follows:

[0058]

[0059] For N triplets in a mini-batch, and Represents the distance between the i-th anchor point and its corresponding positive and negative samples. By adjusting the parameter α, a suitable balance between the gradients of simple samples and hard samples can be found.

[0060] In this embodiment, different weights are adjusted for positive and negative samples, and the predetermined loss function adopts a binomial loss function that balances positive and negative samples. This loss function can better alleviate the problem of imbalance between positive and negative samples. The loss function is as follows:

[0061]

[0062] In the formula, m represents pushing the negative samples away by a margin of m. and represents the cosine similarity between the i-th anchor point and its corresponding positive and negative samples, N p and N n Represent the number of positive and negative sample pairs, α p and α n Represents positive and negative samples being pulled in and pushed away with different weights.

[0063] This step S3 includes the following sub-steps:

[0064] Step S3-1, using a maximum pooling operator along feature channel c to select the most unique object features;

[0065] Step S3-2, use the spatially aware importance generator to generate the position embedding graph P∈R H×W ;

[0066] Step S3-3, in the spatially aware importance generator, two fully connected layers are used to select features f in the prominent layer c ∈R H×W , and calculate the Frobenius inner product of the spatial combination and feature response, and encode and generate an image feature descriptor that suppresses deformation:

[0067] K={k c},c=1,2,……,C

[0068] k c = <f c ,P> F

[0069] In the formula, f c ∈R H×W represents the input feature map of the spatially aware position embedding module in the cth channel, <.,.> represents the Frobenius inner product of the two inputs, and k c is the embedded feature activation of the c-th channel.

[0070] Step S4: In order to improve the robustness of feature representation, multiple spatially aware position embedding modules are used to aggregate and embed the deformation-suppressed image feature descriptors obtained in step S3 to generate the final deformation-suppressed image feature descriptors. Specifically:

[0071] Generate multiple embedding graphs P through multiple spatially aware position embedding modules m ∈R H×W ,m=1,2,……,M, and then encode the input features according to the generated different masks, and connect the embedded features together as the final deformation-suppressed image feature descriptor.

[0072] Step S5: Calculate the sky view factor of the panoramic image using the sky view factor automatic calculation branch.

[0073] Step S5 includes the following sub-steps:

[0074] Step S5-1, using a pyramid scene parsing network to segment the sky, trees, and buildings in the panoramic image, and marking them with white, green, and orange, respectively;

[0075] Step S5-2: the panorama image (x p ,y p ) is used for equiangular fisheye projection, which is then transformed into coordinates (x f ,y f ) and the fisheye image of the sky hemisphere with polar coordinates (r,θ):

[0076]

[0077]

[0078] Step S5-3, divide the sky hemispherical fisheye image into n (default n=36) annular rings, and add the contribution of each annular ring to obtain the sky view factor of the panoramic image:

[0079]

[0080] Where SVF is the sky view factor, is the ratio between the number of pixels in the sky area and the total number of pixels in ring i.

[0081] Step S6, embedding the sky view factor into the final deformation-suppressed image feature descriptor to generate a new image feature descriptor:

[0082] K′=SVF{k c}

[0083] In the formula, k c is the embedded feature activation of the c-th channel.

[0084] Step S7: The spatially aware geolocation neural network performs geolocation based on the new image feature descriptor to obtain the geographic coordinates of the panoramic image.

[0085] As described above, this embodiment provides a spatially-aware geolocation method for panoramic images. The above method can also be configured as a computer program and stored in a computer storage medium. The computer program is used to enable a computer to execute the steps of the above method for spatially-aware geolocation of panoramic images.

[0086] Traditional image geolocation methods refer to comparing and matching captured panoramic images with panoramic images in a database with geographic location information.

[0087] The differences between the spatial perception geolocation method of panoramic images embedded with scene content in this embodiment and the traditional image geolocation method are: first, the geographic coordinates carried by optical remote sensing images with higher resolution, wider coverage and higher accuracy are used for positioning; second, polar coordinate transformation is used to bridge the geometric domain differences between optical remote sensing images and panoramic images; third, a spatial perception framework embedded with scene content is used to suppress image deformation caused by polar coordinate transformation, perceive spatial information and scene content, and make positioning more accurate.

[0088] In this embodiment, in order to verify the effect of this method, two geo-tagged panoramic image and optical remote sensing image data sets are selected to compare with the results obtained by the other two panoramic image geolocation methods in the same data set. Among them, the data set is CVUSA, which contains about 1.5 million geo-tagged ground and aerial image pairs from all over the United States. The experiment collected a total of 35,532 image pairs for training on the CVUSA data set, and collected 8,884 image pairs for testing, and the recall accuracy of the top K positions was used as the evaluation indicator of the experiment. Specifically:

[0089] Given a queried panoramic image, if its corresponding optical remote sensing image is within the nearest top K retrieval images, it is considered "successfully located". The percentage of correctly located query images is reported as r@K. This experiment reports not only the top1% accuracy, but also the top-1, top-5, and top-10 accuracies to better demonstrate the performance of this method.

[0090] Figure 3 It is a comparison chart of experimental evaluation results of the spatial perception geolocation method of panoramic images in an embodiment of the present invention and the traditional geolocation method.

[0091] Table 1 shows the experimental results of the geolocation method of this embodiment and the most advanced positioning method in the prior art on CVUSA data.

[0092]

[0093]

[0094] Table 1

[0095] like Figure 3 As shown in Table 1, the results of geolocation in the CVUSA dataset using the spatially-aware geolocation method for panoramic images of this embodiment all exceed those of the geolocation method in the prior art.

[0096] Among them, the CVM-NET method (deep feature extraction geo-positioning algorithm for panoramic images, excerpted from "Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 7258-7267.") did not consider spatial information and scene content, and only achieved top-1 accuracy of 22.53%, top-5 accuracy of 50.01%, top-10 accuracy of 63.19%, and top-1 accuracy of 93.52% on the CVUSA dataset.

[0097] The Liu&Li method (Panoramic image geo-localization algorithm based on orientation annotation method, excerpted from "Lending orientation to neural networks for cross-view geo-localization [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 5624-5633.") incorporates the orientation information of each pixel into the backbone network for cross-view positioning, achieving top-1 accuracy of 40.79%, top-5 accuracy of 66.82%, top-10 accuracy of 76.36%, and top-1% accuracy of 96.08% on the CVUSA dataset.

[0098] The spatial perception geolocation method for panoramic images in this embodiment uses a polar coordinate transformation method to bridge the geometric domain differences between optical remote sensing images and panoramic images. Due to the use of a spatial perception framework embedded in scene content, image deformation caused by polar coordinate transformation is suppressed, making positioning more accurate. On the CVUSA dataset, the highest accuracies of 92.22% for top-1, 97.97% for top-5, 98.87% for top-10, and 99.79% for top-1% were achieved.

[0099] Figure 4 It is a schematic diagram of the results of verifying the effectiveness of polar coordinate transformation for panoramic image geolocation in an embodiment of the present invention.

[0100] In this embodiment, in order to verify the effectiveness of polar coordinate transformation for panoramic image geolocation, a spatially aware geolocation neural network is trained under two different settings: one uses the original panoramic image and optical remote sensing image, marked as VGG_gp; the other uses the original panoramic image and the optical remote sensing image after polar coordinate transformation, marked as Polar_VGG_gp. Figure 4 As shown, applying polar coordinate transformation to aerial images can greatly improve the performance on the CVUSA dataset.

[0101] Figure 5 It is a schematic diagram of the results of verifying the effectiveness of multiple spatially aware position embedding modules for panoramic image geolocation in an embodiment of the present invention.

[0102] In this embodiment, in order to verify the effectiveness of using multiple spatially aware position embedding modules for aggregated feature embedding, the experiment uses different numbers of spatially aware position embedding modules (denoted as M) for comparison, namely, M=1, M=2, M=4 and M=8. Figure 5As shown in the figure, as M increases, the experimental results can achieve better recall performance. It is worth noting that when M increases from 1 to 2 and from 2 to 4, r@1 is significantly improved (10%), but when M increases from 4 to 8, the experimental results only achieve a small improvement (<4%). Therefore, in this embodiment, the number of spatially aware position embedding modules is set to 8.

[0103] Figure 6 It is a schematic diagram of the results of verifying the effectiveness of the binomial loss function for balancing positive and negative samples for panoramic image geolocation in an embodiment of the present invention.

[0104] In this embodiment, in order to verify the effectiveness of the binomial loss function for balancing positive and negative samples, the experiment trains the spatially aware geolocation neural network of this method under two different settings: one uses the commonly used soft margin weighted triplet loss function to train the network with the spatially aware location embedding module, and the other uses the binomial loss function for balancing positive and negative samples proposed by this method to train the network with the spatially aware location embedding module. In order to save training time, the number of training sets in this experiment is 10,000 pairs of images, and the number of test sets is 2,500 pairs of images. Figure 6 As shown in Figure 2, training the network using a binomial loss function that balances positive and negative samples can effectively improve the performance of the CVUSA dataset.

[0105] Figure 7 It is a schematic diagram of the results of verifying the effectiveness of the sky view factor automatic calculation branch for panoramic image geolocation in an embodiment of the present invention.

[0106] In this embodiment, in order to verify the effectiveness of the automatic calculation branch of the sky view factor, the experiment adopted two different settings: one without adding the automatic calculation branch of the sky view factor; the other with adding the automatic calculation branch of the sky view factor. In addition, both networks use polar coordinate transformation and add a spatial perception position embedding module, and use the binomial loss function that balances positive and negative samples to train the network. In order to save training time, the number of training sets in this experiment is 10,000 pairs of images, and the number of test sets is 2,500 pairs of images. Figure 7 As shown, experiments show that the automatic calculation branch of the sky view factor can greatly improve the performance of the CVUSA dataset.

[0107] Figure 8 Schematic diagram of the structure of the spatial perception geographic positioning system of the panoramic image in the embodiment of the present invention.

[0108] This embodiment also provides a spatially aware geo-positioning system for panoramic images, such as Figure 8As shown, the spatial perception geographic positioning system 100 of the panoramic image includes a geometric domain conversion module 1, a backbone network module 2, a spatial perception feature aggregation module 3, a sky view factor module 4 and an input display module 5.

[0109] The geometric domain conversion module 1 is used to use polar coordinate transformation on the optical remote sensing image data set storing geographic coordinates, so as to align the optical remote sensing image and the panoramic image accordingly.

[0110] The backbone network module 2 is used to train the twin deep neural network based on the binomial loss function of balanced positive and negative samples using the panoramic image dataset and the optical remote sensing image dataset after polar coordinate transformation to generate a spatially aware geolocation network, and obtain the optical remote sensing image and geographic coordinates corresponding to the panoramic image based on the spatially aware geolocation neural network.

[0111] The spatial perception feature aggregation module 3 is used to encode the important features extracted by the spatial perception geographic positioning network and the relative positions of the important features to generate a deformation-suppressing image feature descriptor, and to aggregate and embed the deformation-suppressing image feature descriptor, and connect the embedded features together as the final deformation-suppressing image feature descriptor.

[0112] The sky viewing factor module 4 comprises a panoramic image scene analysis unit, an equiangular fisheye projection unit, and a sky viewing factor numerical calculation unit.

[0113] Among them, the panoramic image scene parsing unit uses a pyramid scene parsing network to segment the sky, trees, and buildings in the panoramic image, and mark them with white, green, and orange respectively.

[0114] The equiangular fisheye projection unit is used to perform equiangular fisheye projection on the segmented panoramic image.

[0115] The sky view factor value calculation unit is used to calculate the sky view factor value of the panoramic image.

[0116] The input display module 5 has a picture storage unit. The input display module 5 is used to display the pictures stored in the picture storage unit for users to input and view.

[0117] Figures 9 to 12 They are respectively a schematic diagram of a picture upload screen, a schematic diagram of a query running window screen, a schematic diagram of a result query screen, and a schematic diagram of a historical query result screen of the spatial-aware geographic positioning system 100 for panoramic images according to an embodiment of the present invention.

[0118] The screen storage unit stores screens for human-computer interaction between the user and the spatial perception geographic positioning system 100 of panoramic images, including a picture upload screen, a query running window screen, a result query screen, and a historical query result screen.

[0119] Among them, Fig. 9 As shown, the picture upload screen is used to display the long upload screen of the panoramic image to be queried when the user queries the geographic coordinates of the panoramic image so that the user can upload the image to be queried.

[0120] like Fig.10 As shown, the query running window screen is used to display to the user the process of performing geographic coordinate query by the spatially aware geographic positioning system 100 of the panoramic image.

[0121] like Fig.11 As shown, the result query screen is used to display the spatial perception geographic positioning system 100 of the panoramic image to the user to perform a geographic coordinate query to obtain the query result, which includes the query time, the name of the file to be queried, the panoramic image to be queried uploaded by the user (the detected panoramic image in the picture), the corresponding optical remote sensing image and the geographic coordinates (longitude and latitude information).

[0122] like Fig.12 As shown, the historical query result screen is used to display the records of query operations for users to select and view.

[0123] Example Function and Effect

[0124] According to the spatial perception geolocation method and system for panoramic images provided by this embodiment, since the optical remote sensing image in the data set is transformed into polar coordinates, the geometric domain difference between it and the panoramic image is bridged. The constructed spatial perception geolocation neural network is an image geolocation neural network that is embedded with spatial perception feature aggregation of scene content, in which each branch of the twin deep neural network can learn the deep features matched between the two diagrams of the panoramic image and the optical remote sensing image without sharing weights; the spatial perception module can encode the extracted important features and the relative positions between the features, thereby generating an image feature descriptor that suppresses deformation and avoids the deformation problem caused by polar coordinate transformation. In addition, since the sky view factor calculation branch in the spatial perception geolocation neural network can parse the scene content of the panoramic image, it effectively pays attention to the spatial position information of the panoramic image, so that the spatial perception geolocation method of the panoramic image of this embodiment has better positioning accuracy. At the same time, since the computational complexity of the spatial perception module is low, the performance of panoramic image geolocation is also improved.

[0125] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A spatially aware geolocation method for panoramic images, characterized in that: The following steps are involved: Step S1, establishing a data set including a panoramic image and an optical remote sensing image storing geographic coordinates, performing polar coordinate conversion on the optical remote sensing image, so that the optical remote sensing image is aligned with the panoramic image; Step S2, constructing a space-aware geolocation neural network, wherein the space-aware geolocation neural network has a backbone network, a space-aware module, and a sky view factor automatic calculation branch; Step S3, using the data set to train the spatially aware geolocation neural network based on a predetermined loss function, applying an attention mechanism to the image feature descriptors generated during the training process, and obtaining image feature descriptors that suppress deformation; Step S4, aggregating and embedding the deformation-suppressed image feature descriptors based on the spatial perception module to generate a final deformation-suppressed image feature descriptor; Step S5, calculating the sky view factor of the panoramic image based on the sky view factor automatic calculation branch; Step S6, embedding the sky view factor into the final deformation-suppressed image feature descriptor to generate a new image feature descriptor; Step S7, the spatially aware geolocation neural network performs geolocation based on the new image feature descriptor to obtain the optical remote sensing image and geographic coordinates corresponding to the panoramic image, Wherein, the step S5 includes the following sub-steps: Step S5-1, using a pyramid scene parsing network to segment the sky, trees, and buildings in the panoramic image, and marking them with white, green, and orange, respectively; Step S5-2: the panorama image (x p ,y p ) is used for equiangular fisheye projection, which is then transformed into coordinates (x f ,y f ) and the fisheye image of the sky hemisphere with polar coordinates (r,θ): Step S5-3, dividing the sky hemispherical fisheye image into n annular rings, and adding the contribution of each annular ring to obtain the sky viewing factor of the panoramic image: Where SVF is the sky view factor, is the ratio between the number of pixels in the sky region and the total number of pixels in ring i.

2. The spatially aware geolocation method of panoramic images according to claim 1, characterized in that: in, In step S1, the size S a ×S a Optical remote sensing images The polar coordinate transformation is performed to obtain a polar coordinate transformation size of H g ×W g The target image 3. The spatially aware geolocation method of panoramic images according to claim 1, characterized in that: in, The step S3 comprises the following sub-steps: Step S3-1, using a maximum pooling operator along feature channel c to select the most unique object features; Step S3-2, use the spatially aware importance generator to generate the position embedding graph P∈R H×W ; Step S3-3, in the spatially aware importance generator, two fully connected layers are used to select features f in the salient layer c ∈R H×W , and calculate the Frobenius inner product of the spatial combination and feature response, and encode and generate the image feature descriptor that suppresses deformation: K={k c },c=1,2,......,C k c =<f c ,P> F In the formula, f c ∈R H×W represents the input feature map of the spatially aware position embedding module in the cth channel, <.,.> represents the Frobenius inner product of two inputs, and k c is the embedded feature activation of the c-th channel.

4. The spatially aware geo-positioning method of panoramic images according to claim 3, characterized in that: in, In step S4, the spatial perception module has a plurality of spatial perception position embedding blocks with the same architecture but different weights, Generate multiple embedding graphs P based on a plurality of the spatially aware position embedding blocks m ∈R H×W , m=1, 2, ...M, and then the input features are encoded according to the generated different masks, and the embedded features are connected together as the final deformation-suppressed image feature descriptor.

5. The spatially aware geolocation method of panoramic images according to claim 1, characterized in that: in, In step S6, the new image feature descriptor is: K′=SVF{k c } Where SVF is the sky view factor, k c is the embedded feature activation of the c-th channel.

6. The spatially aware geolocation method of panoramic images according to claim 1, characterized in that: in, The predetermined loss function is a binomial loss function that balances positive and negative samples: In the formula, m represents pushing the negative samples away by a margin of m. and represents the cosine similarity between the i-th anchor point and its corresponding positive and negative samples, N p and N n Represent the number of positive and negative sample pairs, α p and n p Represents positive and negative samples being pulled in and pushed away with different weights.

7. A spatially aware geo-positioning system for panoramic images, characterized in that: include: Geometric domain conversion module, backbone network module, spatial perception feature aggregation module, sky view factor module and input display module; The geometric domain conversion module is used to perform polar coordinate transformation on the optical remote sensing image data set storing geographic coordinates, so as to align the optical remote sensing image and the panoramic image accordingly. The backbone network module is used to train the twin deep neural network based on the binomial loss function of balanced positive and negative samples using the panoramic image data set and the optical remote sensing image data set after polar coordinate transformation to generate a spatially aware geo-positioning neural network, and obtain the optical remote sensing image and the geographic coordinates corresponding to the panoramic image based on the spatially aware geo-positioning neural network. The spatial perception feature aggregation module is used to encode the important features extracted by the spatial perception geographic positioning network and the relative positions of the important features to generate a deformation-suppressed image feature descriptor, and to aggregate and embed the deformation-suppressed image feature descriptor, and to connect the embedded features together as the final deformation-suppressed image feature descriptor. The sky view factor module comprises a panoramic image scene analysis unit, an equiangular fisheye projection unit and a sky view factor numerical calculation unit. The input display module has a picture storage unit. The screen storage unit has a picture upload screen, a query operation window screen, a result query screen and a historical query result screen. The sky view factor module includes the following sub-steps: Step S5-1, using a pyramid scene parsing network to segment the sky, trees, and buildings in the panoramic image, and marking them with white, green, and orange, respectively; Step S5-2: the panorama image (x p ,y f ) is used for equiangular fisheye projection, which is then transformed into coordinates (x f ,y f ) and the fisheye image of the sky hemisphere with polar coordinates (r,θ): Step S5-3, dividing the sky hemispherical fisheye image into n annular rings, and adding the contribution of each annular ring to obtain the sky viewing factor of the panoramic image: Where SVF is the sky view factor, is the ratio between the number of pixels in the sky region and the total number of pixels in ring i.

8. An electronic device, characterized in that: include: at least one processor and a memory communicatively coupled to the at least one processor; The memory stores instructions. The instructions are executed by the at least one processor to enable the at least one processor to perform the spatially aware geo-positioning method for panoramic images according to any one of claims 1 to 6.

9. A computer-readable storage medium for storing a computer program, characterized in that: The computer program is configured to implement the spatially aware geo-positioning method of panoramic images as claimed in any one of claims 1 to 6 when executed.

Images